2021-11-19T08:21:51.877 INFO:root:teuthology version: 1.1.0-6cfddd5e 2021-11-19T08:21:51.878 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/gabrioux-2021-11-19_08:00:30-rados:cephadm-wip-guits-testing-2021-11-18-1333-pacific-distro-default-smithi/6513316 --description rados:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} --name gabrioux-2021-11-19_08:00:30-rados:cephadm-wip-guits-testing-2021-11-18-1333-pacific-distro-default-smithi --owner scheduled_gabrioux@teuthology --verbose -- /home/teuthworker/archive/gabrioux-2021-11-19_08:00:30-rados:cephadm-wip-guits-testing-2021-11-18-1333-pacific-distro-default-smithi/6513316/orig.config.yaml 2021-11-19T08:21:51.936 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-11-19T08:21:52.023 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2021-11-19_08:00:30-rados:cephadm-wip-guits-testing-2021-11-18-1333-pacific-distro-default-smithi/6513316 branch: wip-guits-testing-2021-11-18-1333-pacific description: rados:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} email: gabrioux@redhat.com first_in_suite: false job_id: '6513316' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: gabrioux-2021-11-19_08:00:30-rados:cephadm-wip-guits-testing-2021-11-18-1333-pacific-distro-default-smithi nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-guits-testing-2021-11-18-1333-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 15m rados osd op timeout: 15m global: mon pg warn min per osd: 0 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: e77c7dee6c987c6680b57de9907bbc4d4962f2b1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: e77c7dee6c987c6680b57de9907bbc4d4962f2b1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-guits-testing-2021-11-18-1333-pacific sha1: e77c7dee6c987c6680b57de9907bbc4d4962f2b1 owner: scheduled_gabrioux@teuthology priority: 80 repo: git://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 sha1: e77c7dee6c987c6680b57de9907bbc4d4962f2b1 sleep_before_teardown: 0 suite: rados:cephadm suite_branch: wip-guits-testing-2021-11-18-1333-pacific suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e77c7dee6c987c6680b57de9907bbc4d4962f2b1/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: e77c7dee6c987c6680b57de9907bbc4d4962f2b1 targets: smithi023.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6yqGo/0OfU+KyNhUaEKneUDOYcM6bkiApZCd6/XzvlVzmn3rDPNGyW+arUa/n7GMW3i1oK0qaJ1IOpW5N0leTHOUfxeTIeWEY/ixYZMrXoSraH6eneFXtrXQlBSGaman0hOKt8eWu3oD+YnKmMwSJRfgWPhmK9fXPUD0D21Imx1F7v0eDtF4pv1hKBNp/hk2609ZvyQrEGqbV1yE36x/ETmf6Dakb9y7tnShDkE2qVTJUKfiT3WyQP2CY9e0FSsBZ+OS9tuLSdWFdt1WbtKpa3Y6adch6GLaWw8iZ8QwgPmc5vMmZiJt4RsHfGFGk+tiyuOpOdEMlv/GkiZEroLKikHVNA9hwBpyKavkPELcbyZjfDUCCLnDqu+DIZOwAjEzw58jrWckiKVeJEtcpuNP40oDoP/hmisS31sNTeRznK3HClPThvMDSDB/te8/9O7zcIwzFd1BDG/HZr1MGYBAm5QufvOBVC9KQLnxR++xuvOvUxlyCIJFkTQqR/dw7s+E= smithi134.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAKnRUf49a1PkCaBCjjKXoFrZVb/3OsMzwWxiupyGvkCA02F1z8GeN8319JgTA+QUd1el9iaQx3mwuf3IR4xSWKzJkNnVj46B6Xf5sLFFa664woSmtIkQuD2SoB8ML7YedXnJNkT/EAuuY+yhCfGx1NXDlgqFtGIhkTyEr49o766KnFkj9xWwIJ5oPgJVjNgrDmhtexOYN1S6JLEfTGwIRb3uZshQTfJ+PEWyIREh71TYZ5oNYOfPnXF/XBqJmiLYax4ee6worcZXgpjgb9fNYth2fdLHO83TF+51RsoulPHwjfzvm+VuaFnS7LtXe3k7IEBtrYoCeeDGTI2uUeEULgAYauwPz9heMdyGiaorlIxMFhbai0ZIo5oj2pSJ/7xJyUwKEYyjH3rXnpGa9nlkj2c/55V6qeUObNPiDqvDnvWVJmAZy6bL/6z0/gXlXIXvAPkYvWYtfOQAwd2cP082nyn9FphWFVT3au65z/Rql9rswDWY3MHNn7c+Eq/8sdgM= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: exclude_packages: - ceph-volume tag: v16.2.4 - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - ceph --format=json mds versions | jq -e ". | add == 4" - fs.pre_upgrade_save: null - kclient: null - print: '**** done kclient client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology_branch: master teuthology_sha1: 6cfddd5e1c7d5ede1367cf3087c21af44eb88356 timestamp: 2021-11-19_08:00:30 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; ceph fs dump; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.20261 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2021-11-19T08:21:52.024 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e77c7dee6c987c6680b57de9907bbc4d4962f2b1/qa; will attempt to use it 2021-11-19T08:21:52.025 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e77c7dee6c987c6680b57de9907bbc4d4962f2b1/qa/tasks 2021-11-19T08:21:52.025 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-11-19T08:21:52.027 INFO:teuthology.task.internal:Checking packages... 2021-11-19T08:21:52.050 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e77c7dee6c987c6680b57de9907bbc4d4962f2b1' 2021-11-19T08:21:52.051 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-11-19T08:21:52.051 INFO:teuthology.packaging:ref: None 2021-11-19T08:21:52.052 INFO:teuthology.packaging:tag: None 2021-11-19T08:21:52.052 INFO:teuthology.packaging:branch: wip-guits-testing-2021-11-18-1333-pacific 2021-11-19T08:21:52.052 INFO:teuthology.packaging:sha1: e77c7dee6c987c6680b57de9907bbc4d4962f2b1 2021-11-19T08:21:52.053 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-guits-testing-2021-11-18-1333-pacific 2021-11-19T08:21:52.295 INFO:teuthology.task.internal:Found packages for ceph version 16.2.6-608.ge77c7dee 2021-11-19T08:21:52.295 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-11-19T08:21:52.305 INFO:teuthology.task.internal:no buildpackages task found 2021-11-19T08:21:52.305 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-11-19T08:21:52.333 INFO:teuthology.task.internal:Saving configuration 2021-11-19T08:21:52.355 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-11-19T08:21:52.364 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-19T08:21:52.386 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2021-11-19_08:00:30-rados:cephadm-wip-guits-testing-2021-11-18-1333-pacific-distro-default-smithi/6513316', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-11-19 08:16:09.930661', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHaSm2sqvgQjUoHfmGpVfDnCkJPit2VTiu9DwPuX0BO2aYu/oB5tR3sNlRicEp+rpNEkQ1d15oeFMhtdmNGpEps='} 2021-11-19T08:21:52.410 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2021-11-19_08:00:30-rados:cephadm-wip-guits-testing-2021-11-18-1333-pacific-distro-default-smithi/6513316', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2021-11-19 08:16:09.929427', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK3YK6r/rko9R11i7Wl2jaqc3sOKWiQoReBRHdDn4MvJZyrxOmwjvl+J/mMJ0FY6ZsQkDFUbJ3u50vVrMzohvYo='} 2021-11-19T08:21:52.410 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-11-19T08:21:52.425 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2021-11-19T08:21:52.425 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2021-11-19T08:21:52.425 INFO:teuthology.run_tasks:Running task console_log... 2021-11-19T08:21:52.511 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcdb4982e18>, signals=[15]) 2021-11-19T08:21:52.512 INFO:teuthology.run_tasks:Running task internal.connect... 2021-11-19T08:21:52.523 INFO:teuthology.task.internal:Opening connections... 2021-11-19T08:21:52.524 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi023.front.sepia.ceph.com 2021-11-19T08:21:52.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-19T08:21:52.597 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2021-11-19T08:21:52.598 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-19T08:21:52.667 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-11-19T08:21:52.677 DEBUG:teuthology.orchestra.run.smithi023:> uname -m 2021-11-19T08:21:52.696 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2021-11-19T08:21:52.697 DEBUG:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2021-11-19T08:21:52.756 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="CentOS Stream" 2021-11-19T08:21:52.757 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="8" 2021-11-19T08:21:52.757 INFO:teuthology.orchestra.run.smithi023.stdout:ID="centos" 2021-11-19T08:21:52.758 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE="rhel fedora" 2021-11-19T08:21:52.758 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="8" 2021-11-19T08:21:52.758 INFO:teuthology.orchestra.run.smithi023.stdout:PLATFORM_ID="platform:el8" 2021-11-19T08:21:52.759 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-19T08:21:52.759 INFO:teuthology.orchestra.run.smithi023.stdout:ANSI_COLOR="0;31" 2021-11-19T08:21:52.759 INFO:teuthology.orchestra.run.smithi023.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-19T08:21:52.760 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://centos.org/" 2021-11-19T08:21:52.760 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-19T08:21:52.760 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-19T08:21:52.761 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-19T08:21:52.762 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2021-11-19T08:21:52.781 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2021-11-19T08:21:52.799 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2021-11-19T08:21:52.799 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2021-11-19T08:21:52.856 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2021-11-19T08:21:52.856 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="8" 2021-11-19T08:21:52.856 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2021-11-19T08:21:52.856 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2021-11-19T08:21:52.857 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="8" 2021-11-19T08:21:52.857 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el8" 2021-11-19T08:21:52.857 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 8" 2021-11-19T08:21:52.857 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2021-11-19T08:21:52.858 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-11-19T08:21:52.858 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2021-11-19T08:21:52.858 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-11-19T08:21:52.858 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-11-19T08:21:52.859 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-11-19T08:21:52.859 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2021-11-19T08:21:52.876 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-11-19T08:21:52.888 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-11-19T08:21:52.910 INFO:teuthology.task.internal:Checking for old test directory... 2021-11-19T08:21:52.911 DEBUG:teuthology.orchestra.run.smithi023:> test '!' -e /home/ubuntu/cephtest 2021-11-19T08:21:52.914 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2021-11-19T08:21:52.930 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-11-19T08:21:52.942 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-11-19T08:21:52.942 DEBUG:teuthology.orchestra.run.smithi023:> test -z $(ls -A /var/lib/ceph) 2021-11-19T08:21:52.972 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2021-11-19T08:21:52.994 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-11-19T08:21:53.052 INFO:teuthology.run_tasks:Running task kernel... 2021-11-19T08:21:53.073 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2021-11-19T08:21:53.074 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-11-19T08:21:53.074 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2021-11-19T08:21:53.075 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2021-11-19T08:21:53.092 INFO:teuthology.orchestra.run.smithi134.stdout:4.18.0-301.1.el8.x86_64 2021-11-19T08:21:53.092 INFO:teuthology.task.kernel:Running kernel on smithi134: 4.18.0-301.1.el8.x86_64 2021-11-19T08:21:53.093 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2021-11-19T08:21:53.094 INFO:teuthology.orchestra.run.smithi023.stdout:4.18.0-301.1.el8.x86_64 2021-11-19T08:21:53.094 INFO:teuthology.task.kernel:Running kernel on smithi023: 4.18.0-301.1.el8.x86_64 2021-11-19T08:21:53.094 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum install -y kernel 2021-11-19T08:21:55.707 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-8 - AppStream 49 MB/s | 17 MB 00:00 2021-11-19T08:21:55.829 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-8 - AppStream 41 MB/s | 17 MB 00:00 2021-11-19T08:21:59.101 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-8 - Base 45 MB/s | 15 MB 00:00 2021-11-19T08:21:59.147 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-8 - Base 40 MB/s | 15 MB 00:00 2021-11-19T08:22:01.322 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-8 - Extras 236 kB/s | 16 kB 00:00 2021-11-19T08:22:01.498 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-8 - Extras 243 kB/s | 16 kB 00:00 2021-11-19T08:22:02.416 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 11 MB/s | 11 MB 00:00 2021-11-19T08:22:03.295 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 6.4 MB/s | 11 MB 00:01 2021-11-19T08:22:04.820 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2021-11-19T08:22:05.874 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2021-11-19T08:22:06.816 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-4.18.0-240.22.1.el8_3.x86_64 is already installed. 2021-11-19T08:22:06.816 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-4.18.0-301.1.el8.x86_64 is already installed. 2021-11-19T08:22:06.817 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-11-19T08:22:07.143 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:22:07.144 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:22:07.145 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2021-11-19T08:22:07.145 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:22:07.145 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:22:07.145 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 7.0 M 2021-11-19T08:22:07.146 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 38 M 2021-11-19T08:22:07.146 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 30 M 2021-11-19T08:22:07.146 INFO:teuthology.orchestra.run.smithi134.stdout:Removing: 2021-11-19T08:22:07.146 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 4.18.0-240.el8 @anaconda 0 2021-11-19T08:22:07.147 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 4.18.0-240.el8 @anaconda 62 M 2021-11-19T08:22:07.147 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 4.18.0-240.el8 @anaconda 21 M 2021-11-19T08:22:07.147 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:22:07.147 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:22:07.147 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:22:07.148 INFO:teuthology.orchestra.run.smithi134.stdout:Install 3 Packages 2021-11-19T08:22:07.148 INFO:teuthology.orchestra.run.smithi134.stdout:Remove 3 Packages 2021-11-19T08:22:07.149 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:22:07.149 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 74 M 2021-11-19T08:22:07.149 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:22:07.291 INFO:teuthology.orchestra.run.smithi134.stdout:(1/3): kernel-4.18.0-348.2.1.el8_5.x86_64.rpm 48 MB/s | 7.0 MB 00:00 2021-11-19T08:22:07.976 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-4.18.0-240.22.1.el8_3.x86_64 is already installed. 2021-11-19T08:22:07.976 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-4.18.0-301.1.el8.x86_64 is already installed. 2021-11-19T08:22:07.977 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-11-19T08:22:08.205 INFO:teuthology.orchestra.run.smithi134.stdout:(2/3): kernel-core-4.18.0-348.2.1.el8_5.x86_64. 35 MB/s | 38 MB 00:01 2021-11-19T08:22:08.315 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:22:08.317 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:22:08.317 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2021-11-19T08:22:08.317 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:22:08.317 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:22:08.318 INFO:teuthology.orchestra.run.smithi023.stdout: kernel x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 7.0 M 2021-11-19T08:22:08.318 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-core x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 38 M 2021-11-19T08:22:08.318 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules x86_64 4.18.0-348.2.1.el8_5 CentOS-Base 30 M 2021-11-19T08:22:08.318 INFO:teuthology.orchestra.run.smithi023.stdout:Removing: 2021-11-19T08:22:08.319 INFO:teuthology.orchestra.run.smithi023.stdout: kernel x86_64 4.18.0-240.el8 @anaconda 0 2021-11-19T08:22:08.319 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-core x86_64 4.18.0-240.el8 @anaconda 62 M 2021-11-19T08:22:08.319 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules x86_64 4.18.0-240.el8 @anaconda 21 M 2021-11-19T08:22:08.319 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:22:08.319 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:22:08.320 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:22:08.320 INFO:teuthology.orchestra.run.smithi023.stdout:Install 3 Packages 2021-11-19T08:22:08.320 INFO:teuthology.orchestra.run.smithi023.stdout:Remove 3 Packages 2021-11-19T08:22:08.320 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:22:08.321 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 74 M 2021-11-19T08:22:08.321 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:22:08.424 INFO:teuthology.orchestra.run.smithi134.stdout:(3/3): kernel-modules-4.18.0-348.2.1.el8_5.x86_ 23 MB/s | 30 MB 00:01 2021-11-19T08:22:08.428 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:22:08.429 INFO:teuthology.orchestra.run.smithi134.stdout:Total 58 MB/s | 74 MB 00:01 2021-11-19T08:22:08.476 INFO:teuthology.orchestra.run.smithi023.stdout:(1/3): kernel-4.18.0-348.2.1.el8_5.x86_64.rpm 44 MB/s | 7.0 MB 00:00 2021-11-19T08:22:08.561 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:22:09.230 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:22:09.230 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:22:09.389 INFO:teuthology.orchestra.run.smithi023.stdout:(2/3): kernel-core-4.18.0-348.2.1.el8_5.x86_64. 35 MB/s | 38 MB 00:01 2021-11-19T08:22:09.607 INFO:teuthology.orchestra.run.smithi023.stdout:(3/3): kernel-modules-4.18.0-348.2.1.el8_5.x86_ 23 MB/s | 30 MB 00:01 2021-11-19T08:22:09.611 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:22:09.611 INFO:teuthology.orchestra.run.smithi023.stdout:Total 57 MB/s | 74 MB 00:01 2021-11-19T08:22:09.749 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:22:10.341 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:22:10.358 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:22:10.405 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:22:10.405 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:22:11.537 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:22:11.542 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:22:13.020 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:22:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-19T08:22:14.360 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:22:14.805 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-19T08:22:17.745 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-19T08:22:18.130 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-19T08:22:19.134 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-19T08:22:19.487 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-19T08:22:21.188 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-19T08:22:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-19T08:22:21.359 INFO:teuthology.orchestra.run.smithi134.stdout: Erasing : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-19T08:22:21.581 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-4.18.0-240.el8.x86_64 4/6 2021-11-19T08:22:21.581 INFO:teuthology.orchestra.run.smithi134.stdout: Erasing : kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-19T08:22:22.778 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-19T08:22:22.984 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-19T08:22:22.984 INFO:teuthology.orchestra.run.smithi023.stdout: Erasing : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-19T08:22:23.209 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-4.18.0-240.el8.x86_64 4/6 2021-11-19T08:22:23.209 INFO:teuthology.orchestra.run.smithi023.stdout: Erasing : kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-19T08:22:23.728 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-19T08:22:23.944 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-19T08:22:23.954 INFO:teuthology.orchestra.run.smithi134.stdout: Erasing : kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-19T08:22:25.596 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-4.18.0-240.el8.x86_64 5/6 2021-11-19T08:22:25.813 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-19T08:22:25.823 INFO:teuthology.orchestra.run.smithi023.stdout: Erasing : kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-19T08:22:26.889 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-19T08:22:29.075 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-19T08:22:47.486 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 6/6 2021-11-19T08:22:47.870 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-19T08:22:47.871 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-19T08:22:47.871 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-19T08:22:47.871 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-19T08:22:47.871 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-19T08:22:47.872 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-4.18.0-240.el8.x86_64 5/6 2021-11-19T08:22:48.372 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-4.18.0-240.el8.x86_64 6/6 2021-11-19T08:22:48.372 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:22:48.373 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:22:48.373 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:48.373 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core-4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:48.373 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:48.374 INFO:teuthology.orchestra.run.smithi134.stdout:Removed: 2021-11-19T08:22:48.374 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-4.18.0-240.el8.x86_64 kernel-core-4.18.0-240.el8.x86_64 2021-11-19T08:22:48.374 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-4.18.0-240.el8.x86_64 2021-11-19T08:22:48.374 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:22:48.375 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:22:48.503 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel --last | head -n 1 2021-11-19T08:22:48.659 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Fri 19 Nov 2021 08:22:21 AM UTC 2021-11-19T08:22:48.660 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:48.660 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-301.1.el8.x86_64 Expected: 4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:48.661 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-11-19T08:22:48.661 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-11-19T08:22:48.661 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2021-11-19T08:22:48.661 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2021-11-19T08:22:48.662 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel --last | head -n 1 2021-11-19T08:22:48.805 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Fri 19 Nov 2021 08:22:21 AM UTC 2021-11-19T08:22:48.807 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:48.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2021-11-19T08:22:48.838 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2021-11-19T08:22:48.839 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2021-11-19T08:22:48.839 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.02 2021-11-19T08:22:48.839 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 99.el8 2021-11-19T08:22:48.839 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2021-11-19T08:22:48.840 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Thu 13 May 2021 06:08:01 PM UTC 2021-11-19T08:22:48.840 INFO:teuthology.orchestra.run.smithi134.stdout:Group : System Environment/Base 2021-11-19T08:22:48.840 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 9264704 2021-11-19T08:22:48.841 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2021-11-19T08:22:48.841 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 09:11:55 PM UTC, Key ID 05b555b38483c65d 2021-11-19T08:22:48.841 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2021-11-19T08:22:48.841 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Tue 02 Mar 2021 09:03:27 PM UTC 2021-11-19T08:22:48.842 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : kbuilder.bsys.centos.org 2021-11-19T08:22:48.842 INFO:teuthology.orchestra.run.smithi134.stdout:Relocations : (not relocatable) 2021-11-19T08:22:48.842 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : CentOS BuildSystem 2021-11-19T08:22:48.842 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2021-11-19T08:22:48.843 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2021-11-19T08:22:48.843 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2021-11-19T08:22:48.843 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2021-11-19T08:22:48.843 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:22:48.843 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-11-19T08:22:48.844 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2021-11-19T08:22:48.844 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2021-11-19T08:22:48.844 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2021-11-19T08:22:48.844 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:22:48.845 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2021-11-19T08:22:48.846 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-11-19T08:22:48.846 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:48.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-11-19T08:22:50.227 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2021-11-19T08:22:51.052 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-4.18.0-348.2.1.el8_5.x86_64 6/6 2021-11-19T08:22:51.454 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-4.18.0-240.el8.x86_64 6/6 2021-11-19T08:22:51.455 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-4.18.0-348.2.1.el8_5.x86_64 1/6 2021-11-19T08:22:51.455 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-core-4.18.0-348.2.1.el8_5.x86_64 2/6 2021-11-19T08:22:51.455 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-modules-4.18.0-348.2.1.el8_5.x86_64 3/6 2021-11-19T08:22:51.456 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-4.18.0-240.el8.x86_64 4/6 2021-11-19T08:22:51.456 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-core-4.18.0-240.el8.x86_64 5/6 2021-11-19T08:22:52.363 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-modules-4.18.0-240.el8.x86_64 6/6 2021-11-19T08:22:52.363 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:22:52.363 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:22:52.364 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:52.364 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-core-4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:52.364 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:52.364 INFO:teuthology.orchestra.run.smithi023.stdout:Removed: 2021-11-19T08:22:52.365 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-4.18.0-240.el8.x86_64 kernel-core-4.18.0-240.el8.x86_64 2021-11-19T08:22:52.365 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-4.18.0-240.el8.x86_64 2021-11-19T08:22:52.365 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:22:52.365 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:22:52.477 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q kernel --last | head -n 1 2021-11-19T08:22:52.651 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Fri 19 Nov 2021 08:22:22 AM UTC 2021-11-19T08:22:52.652 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:52.653 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-301.1.el8.x86_64 Expected: 4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:52.653 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-11-19T08:22:52.653 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-11-19T08:22:52.653 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2021-11-19T08:22:52.654 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi023.front.sepia.ceph.com, path=None, version=distro) 2021-11-19T08:22:52.654 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q kernel --last | head -n 1 2021-11-19T08:22:52.774 INFO:teuthology.orchestra.run.smithi134.stderr:done 2021-11-19T08:22:52.776 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2021-11-19T08:22:52.802 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-4.18.0-348.2.1.el8_5.x86_64 Fri 19 Nov 2021 08:22:22 AM UTC 2021-11-19T08:22:52.803 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:52.803 DEBUG:teuthology.orchestra.run.smithi023:> sudo rpm -qi grub2-tools 2021-11-19T08:22:52.807 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.oqY6DtZAF7 2021-11-19T08:22:52.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oqY6DtZAF7 2021-11-19T08:22:52.834 INFO:teuthology.orchestra.run.smithi023.stdout:Name : grub2-tools 2021-11-19T08:22:52.835 INFO:teuthology.orchestra.run.smithi023.stdout:Epoch : 1 2021-11-19T08:22:52.835 INFO:teuthology.orchestra.run.smithi023.stdout:Version : 2.02 2021-11-19T08:22:52.835 INFO:teuthology.orchestra.run.smithi023.stdout:Release : 99.el8 2021-11-19T08:22:52.836 INFO:teuthology.orchestra.run.smithi023.stdout:Architecture: x86_64 2021-11-19T08:22:52.836 INFO:teuthology.orchestra.run.smithi023.stdout:Install Date: Thu 13 May 2021 06:08:01 PM UTC 2021-11-19T08:22:52.836 INFO:teuthology.orchestra.run.smithi023.stdout:Group : System Environment/Base 2021-11-19T08:22:52.836 INFO:teuthology.orchestra.run.smithi023.stdout:Size : 9264704 2021-11-19T08:22:52.837 INFO:teuthology.orchestra.run.smithi023.stdout:License : GPLv3+ 2021-11-19T08:22:52.837 INFO:teuthology.orchestra.run.smithi023.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 09:11:55 PM UTC, Key ID 05b555b38483c65d 2021-11-19T08:22:52.837 INFO:teuthology.orchestra.run.smithi023.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2021-11-19T08:22:52.837 INFO:teuthology.orchestra.run.smithi023.stdout:Build Date : Tue 02 Mar 2021 09:03:27 PM UTC 2021-11-19T08:22:52.838 INFO:teuthology.orchestra.run.smithi023.stdout:Build Host : kbuilder.bsys.centos.org 2021-11-19T08:22:52.838 INFO:teuthology.orchestra.run.smithi023.stdout:Relocations : (not relocatable) 2021-11-19T08:22:52.838 INFO:teuthology.orchestra.run.smithi023.stdout:Packager : CentOS BuildSystem 2021-11-19T08:22:52.838 INFO:teuthology.orchestra.run.smithi023.stdout:Vendor : CentOS 2021-11-19T08:22:52.839 INFO:teuthology.orchestra.run.smithi023.stdout:URL : http://www.gnu.org/software/grub/ 2021-11-19T08:22:52.839 INFO:teuthology.orchestra.run.smithi023.stdout:Summary : Support tools for GRUB. 2021-11-19T08:22:52.840 INFO:teuthology.orchestra.run.smithi023.stdout:Description : 2021-11-19T08:22:52.840 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:22:52.840 INFO:teuthology.orchestra.run.smithi023.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-11-19T08:22:52.841 INFO:teuthology.orchestra.run.smithi023.stdout:customizable bootloader with modular architecture. It supports a rich 2021-11-19T08:22:52.841 INFO:teuthology.orchestra.run.smithi023.stdout:variety of kernel formats, file systems, computer architectures and 2021-11-19T08:22:52.841 INFO:teuthology.orchestra.run.smithi023.stdout:hardware devices. 2021-11-19T08:22:52.842 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:22:52.842 INFO:teuthology.orchestra.run.smithi023.stdout:This subpackage provides tools for support of all platforms. 2021-11-19T08:22:52.843 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-11-19T08:22:52.844 INFO:teuthology.task.kernel:Updating grub on smithi023 to boot 4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:52.845 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-11-19T08:22:52.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.oqY6DtZAF7 2021-11-19T08:22:53.032 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.oqY6DtZAF7 is 6KB 2021-11-19T08:22:53.082 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.oqY6DtZAF7 2021-11-19T08:22:53.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2021-11-19T08:22:53.164 INFO:teuthology.orchestra.run.smithi134.stdout:3331cc438d4e47d3b31025a6d3bd39f2-0-rescue.conf 2021-11-19T08:22:53.165 INFO:teuthology.orchestra.run.smithi134.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-240.22.1.el8_3.x86_64.conf 2021-11-19T08:22:53.165 INFO:teuthology.orchestra.run.smithi134.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-301.1.el8.x86_64.conf 2021-11-19T08:22:53.165 INFO:teuthology.orchestra.run.smithi134.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64.conf 2021-11-19T08:22:53.166 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default 3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:53.360 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2021-11-19T08:22:54.131 INFO:teuthology.orchestra.run.smithi023.stderr:Generating grub configuration file ... 2021-11-19T08:22:56.240 INFO:teuthology.orchestra.run.smithi023.stderr:done 2021-11-19T08:22:56.242 DEBUG:teuthology.orchestra.run.smithi023:> mktemp 2021-11-19T08:22:56.260 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.55waACPUxn 2021-11-19T08:22:56.260 DEBUG:teuthology.orchestra.run.smithi023:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.55waACPUxn 2021-11-19T08:22:56.331 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.55waACPUxn 2021-11-19T08:22:56.478 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.55waACPUxn is 6KB 2021-11-19T08:22:56.528 DEBUG:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.55waACPUxn 2021-11-19T08:22:56.544 DEBUG:teuthology.orchestra.run.smithi023:> sudo /bin/ls /boot/loader/entries || true 2021-11-19T08:22:56.607 INFO:teuthology.orchestra.run.smithi023.stdout:3331cc438d4e47d3b31025a6d3bd39f2-0-rescue.conf 2021-11-19T08:22:56.608 INFO:teuthology.orchestra.run.smithi023.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-240.22.1.el8_3.x86_64.conf 2021-11-19T08:22:56.608 INFO:teuthology.orchestra.run.smithi023.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-301.1.el8.x86_64.conf 2021-11-19T08:22:56.608 INFO:teuthology.orchestra.run.smithi023.stdout:3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64.conf 2021-11-19T08:22:56.610 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-set-default 3331cc438d4e47d3b31025a6d3bd39f2-4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:22:56.775 DEBUG:teuthology.orchestra.run.smithi023:> sudo shutdown -r now 2021-11-19T08:23:23.387 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2021-11-19T08:23:23.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2021-11-19T08:23:23.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-19T08:23:26.778 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2021-11-19T08:23:26.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2021-11-19T08:23:26.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-19T08:24:41.827 DEBUG:teuthology.orchestra.remote:timed out 2021-11-19T08:24:45.107 DEBUG:teuthology.orchestra.remote:timed out 2021-11-19T08:25:11.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2021-11-19T08:25:11.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-19T08:25:12.259 DEBUG:teuthology.orchestra.run.smithi134:> true 2021-11-19T08:25:12.532 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2021-11-19T08:25:12.533 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-348.2.1.el8_5.x86_64"... 2021-11-19T08:25:12.533 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2021-11-19T08:25:12.550 INFO:teuthology.orchestra.run.smithi134.stdout:4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:25:12.551 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.2.1.el8_5.x86_64 vs 4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:25:12.551 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-11-19T08:25:12.551 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-11-19T08:25:13.552 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2021-11-19T08:25:13.553 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-11-19T08:25:13.650 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2021-11-19T08:25:13.662 DEBUG:teuthology.parallel:result is None 2021-11-19T08:25:15.109 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2021-11-19T08:25:15.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-19T08:25:15.601 DEBUG:teuthology.orchestra.run.smithi023:> true 2021-11-19T08:25:15.873 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2021-11-19T08:25:15.873 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-348.2.1.el8_5.x86_64"... 2021-11-19T08:25:15.873 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2021-11-19T08:25:15.929 INFO:teuthology.orchestra.run.smithi023.stdout:4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:25:15.931 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.2.1.el8_5.x86_64 vs 4.18.0-348.2.1.el8_5.x86_64 2021-11-19T08:25:15.931 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-11-19T08:25:15.931 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-11-19T08:25:16.932 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2021-11-19T08:25:16.932 DEBUG:teuthology.orchestra.run.smithi023:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-11-19T08:25:17.036 INFO:teuthology.orchestra.run.smithi023.stdout:ttyS1 2021-11-19T08:25:17.054 DEBUG:teuthology.parallel:result is None 2021-11-19T08:25:17.055 INFO:teuthology.run_tasks:Running task internal.base... 2021-11-19T08:25:17.066 INFO:teuthology.task.internal:Creating test directory... 2021-11-19T08:25:17.067 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-11-19T08:25:17.070 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-11-19T08:25:17.089 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-11-19T08:25:17.100 INFO:teuthology.run_tasks:Running task internal.archive... 2021-11-19T08:25:17.127 INFO:teuthology.task.internal:Creating archive directory... 2021-11-19T08:25:17.128 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-11-19T08:25:17.130 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-11-19T08:25:17.164 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-11-19T08:25:17.174 INFO:teuthology.task.internal:Enabling coredump saving... 2021-11-19T08:25:17.175 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-11-19T08:25:17.197 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-11-19T08:25:17.232 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-19T08:25:17.235 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-19T08:25:17.243 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-19T08:25:17.246 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-11-19T08:25:17.248 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-11-19T08:25:17.258 INFO:teuthology.task.internal:Configuring sudo... 2021-11-19T08:25:17.259 DEBUG:teuthology.orchestra.run.smithi023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-11-19T08:25:17.291 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-11-19T08:25:17.318 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-11-19T08:25:17.331 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-11-19T08:25:17.332 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-11-19T08:25:17.359 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-11-19T08:25:17.376 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-19T08:25:17.456 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-19T08:25:17.571 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-19T08:25:17.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-19T08:25:17.658 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2021-11-19T08:25:17.658 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-11-19T08:25:17.722 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-19T08:25:17.762 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-11-19T08:25:17.815 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-19T08:25:17.895 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-11-19T08:25:17.939 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2021-11-19T08:25:17.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-11-19T08:25:18.004 DEBUG:teuthology.orchestra.run.smithi023:> sudo service rsyslog restart 2021-11-19T08:25:18.006 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2021-11-19T08:25:18.061 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-19T08:25:18.093 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-19T08:25:18.553 INFO:teuthology.run_tasks:Running task internal.timer... 2021-11-19T08:25:18.571 INFO:teuthology.task.internal:Starting timer... 2021-11-19T08:25:18.571 INFO:teuthology.run_tasks:Running task pcp... 2021-11-19T08:25:18.614 INFO:teuthology.run_tasks:Running task selinux... 2021-11-19T08:25:18.645 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-11-19T08:25:18.645 DEBUG:teuthology.orchestra.run.smithi023:> sudo service auditd rotate 2021-11-19T08:25:18.704 INFO:teuthology.orchestra.run.smithi023.stdout:Rotating logs: [ OK ] 2021-11-19T08:25:18.706 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2021-11-19T08:25:18.749 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: [ OK ] 2021-11-19T08:25:18.750 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-11-19T08:25:18.751 DEBUG:teuthology.orchestra.run.smithi023:> /usr/sbin/getenforce 2021-11-19T08:25:18.776 INFO:teuthology.orchestra.run.smithi023.stdout:Permissive 2021-11-19T08:25:18.776 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2021-11-19T08:25:18.808 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2021-11-19T08:25:18.809 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi023.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi134.front.sepia.ceph.com': 'permissive'} 2021-11-19T08:25:18.809 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-19T08:25:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-19T08:25:18.842 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-19T08:25:18.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-19T08:25:18.874 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-11-19T08:25:18.874 DEBUG:teuthology.orchestra.run.smithi023:> sudo /usr/sbin/setenforce permissive 2021-11-19T08:25:18.922 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2021-11-19T08:25:18.953 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-11-19T08:25:18.963 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-11-19T08:25:18.964 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-11-19T08:25:18.985 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-11-19T08:25:18.987 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi023.front.sepia.ceph.com,smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-11-19T08:25:19.393 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-11-19T08:25:20.228 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-11-19T08:25:23.157 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-11-19T08:25:23.280 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:23 +0000 (0:00:00.139) 0:00:00.139 ******* 2021-11-19T08:25:23.292 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:23 +0000 (0:00:00.012) 0:00:00.151 ******* 2021-11-19T08:25:24.701 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi134.front.sepia.ceph.com] 2021-11-19T08:25:24.738 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] 2021-11-19T08:25:24.772 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:24 +0000 (0:00:01.479) 0:00:01.631 ******* 2021-11-19T08:25:24.800 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:24 +0000 (0:00:00.027) 0:00:01.659 ******* 2021-11-19T08:25:25.035 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-11-19T08:25:25.036 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-19T08:25:25.056 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:25 +0000 (0:00:00.256) 0:00:01.915 ******* 2021-11-19T08:25:25.063 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-19T08:25:25.091 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:25 +0000 (0:00:00.034) 0:00:01.949 ******* 2021-11-19T08:25:25.569 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-11-19T08:25:25.571 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-11-19T08:25:25.595 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:25 +0000 (0:00:00.504) 0:00:02.454 ******* 2021-11-19T08:25:25.602 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-11-19T08:25:25.629 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:25 +0000 (0:00:00.033) 0:00:02.487 ******* 2021-11-19T08:25:25.657 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-11-19T08:25:25.659 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:25:25.681 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:25 +0000 (0:00:00.052) 0:00:02.540 ******* 2021-11-19T08:25:25.699 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:25:25.722 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:25 +0000 (0:00:00.040) 0:00:02.581 ******* 2021-11-19T08:25:25.950 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-11-19T08:25:25.952 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-19T08:25:25.972 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:25 +0000 (0:00:00.250) 0:00:02.831 ******* 2021-11-19T08:25:25.992 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:25 +0000 (0:00:00.018) 0:00:02.850 ******* 2021-11-19T08:25:26.715 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-11-19T08:25:26.717 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310326.0406327-23437-37371803727719/source", "state": "file", "uid": 0} changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310326.0162516-23433-188999336855234/source", "state": "file", "uid": 0} 2021-11-19T08:25:26.737 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:26 +0000 (0:00:00.745) 0:00:03.596 ******* 2021-11-19T08:25:26.756 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:26 +0000 (0:00:00.019) 0:00:03.615 ******* 2021-11-19T08:25:27.274 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-11-19T08:25:27.276 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-11-19T08:25:27.278 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail 2021-11-19T08:25:27.280 INFO:teuthology.task.ansible.out:.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@ 2021-11-19T08:25:27.282 INFO:teuthology.task.ansible.out:aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-11-19T08:25:27.358 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** 2021-11-19T08:25:27.360 INFO:teuthology.task.ansible.out:skipping: no hosts matched PLAY [testnodes] *************************************************************** 2021-11-19T08:25:27.372 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Friday 19 November 2021 08:25:27 +0000 (0:00:00.615) 0:00:04.231 ******* 2021-11-19T08:25:27.903 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] 2021-11-19T08:25:27.911 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] 2021-11-19T08:25:27.928 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** Friday 19 November 2021 08:25:27 +0000 (0:00:00.555) 0:00:04.787 ******* 2021-11-19T08:25:27.953 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-11-19T08:25:27.964 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-11-19T08:25:27.970 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-11-19T08:25:28.151 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:28 +0000 (0:00:00.222) 0:00:05.010 ******* 2021-11-19T08:25:28.172 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:28 +0000 (0:00:00.021) 0:00:05.031 ******* 2021-11-19T08:25:28.792 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi023.front.sepia.ceph.com] 2021-11-19T08:25:28.822 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] 2021-11-19T08:25:28.850 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-11-19T08:25:28.852 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:28 +0000 (0:00:00.678) 0:00:05.709 ******* 2021-11-19T08:25:28.894 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:28 +0000 (0:00:00.042) 0:00:05.752 ******* 2021-11-19T08:25:29.493 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-11-19T08:25:29.494 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.169933", "end": "2021-11-19 08:25:29.465745", "rc": 0, "start": "2021-11-19 08:25:29.295812", "stderr": "", "stderr_lines": [], "stdout": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "stdout_lines": ["d72e06d3c78c8d9639a8ca75effa043ee468910f"]} 2021-11-19T08:25:29.515 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:29 +0000 (0:00:00.621) 0:00:06.374 ******* 2021-11-19T08:25:29.521 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.190589", "end": "2021-11-19 08:25:29.486436", "rc": 0, "start": "2021-11-19 08:25:29.295847", "stderr": "", "stderr_lines": [], "stdout": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "stdout_lines": ["d72e06d3c78c8d9639a8ca75effa043ee468910f"]} 2021-11-19T08:25:29.546 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:29 +0000 (0:00:00.030) 0:00:06.404 ******* 2021-11-19T08:25:29.718 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-11-19T08:25:29.720 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013151", "end": "2021-11-19 08:25:29.713865", "failed_when_result": false, "rc": 0, "start": "2021-11-19 08:25:29.700714", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} 2021-11-19T08:25:29.741 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:29 +0000 (0:00:00.195) 0:00:06.600 ******* 2021-11-19T08:25:29.760 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.008745", "end": "2021-11-19 08:25:29.744104", "failed_when_result": false, "rc": 0, "start": "2021-11-19 08:25:29.735359", "stderr": "", "stderr_lines": [], "stdout": "2f6366fd31821994f3d217ad18c1675bf42a2115", "stdout_lines": ["2f6366fd31821994f3d217ad18c1675bf42a2115"]} 2021-11-19T08:25:29.784 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:29 +0000 (0:00:00.042) 0:00:06.643 ******* 2021-11-19T08:25:29.812 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-11-19T08:25:29.813 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:25:29.835 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:29 +0000 (0:00:00.051) 0:00:06.694 ******* 2021-11-19T08:25:29.863 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:25:29.886 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:29 +0000 (0:00:00.051) 0:00:06.745 ******* 2021-11-19T08:25:29.904 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:25:29.927 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:29 +0000 (0:00:00.040) 0:00:06.786 ******* 2021-11-19T08:25:29.954 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:25:29.980 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:29 +0000 (0:00:00.052) 0:00:06.839 ******* 2021-11-19T08:25:29.998 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-11-19T08:25:29.999 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:25:30.022 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:30 +0000 (0:00:00.041) 0:00:06.881 ******* 2021-11-19T08:25:30.050 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:25:30.081 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:30 +0000 (0:00:00.059) 0:00:06.940 ******* 2021-11-19T08:25:30.098 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-11-19T08:25:30.100 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:25:30.124 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:30 +0000 (0:00:00.042) 0:00:06.983 ******* 2021-11-19T08:25:30.152 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:25:30.176 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:30 +0000 (0:00:00.051) 0:00:07.035 ******* 2021-11-19T08:25:30.204 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-11-19T08:25:30.206 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:25:30.227 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:30 +0000 (0:00:00.051) 0:00:07.086 ******* 2021-11-19T08:25:30.244 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:25:30.267 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:30 +0000 (0:00:00.040) 0:00:07.126 ******* 2021-11-19T08:25:30.461 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-11-19T08:25:30.463 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-19T08:25:30.484 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:30 +0000 (0:00:00.217) 0:00:07.343 ******* 2021-11-19T08:25:30.512 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-19T08:25:30.538 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:25:30 +0000 (0:00:00.053) 0:00:07.397 ******* 2021-11-19T08:25:30.762 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-11-19T08:25:30.764 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-11-19T08:25:30.836 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-11-19T08:25:30.961 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-11-19T08:25:31.035 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-11-19T08:25:31.160 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-11-19T08:25:31.244 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-11-19T08:25:31.348 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-11-19T08:25:31.453 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-11-19T08:25:31.547 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-11-19T08:25:31.652 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-11-19T08:25:31.746 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-11-19T08:25:31.861 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-11-19T08:25:31.935 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-11-19T08:25:32.060 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-11-19T08:25:32.133 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-11-19T08:25:32.270 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-11-19T08:25:32.334 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-11-19T08:25:32.471 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-11-19T08:25:32.535 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-11-19T08:25:32.690 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-11-19T08:25:32.722 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-11-19T08:25:32.889 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-11-19T08:25:32.931 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-11-19T08:25:33.099 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-11-19T08:25:33.121 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-11-19T08:25:33.306 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-11-19T08:25:33.328 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-11-19T08:25:33.506 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-11-19T08:25:33.528 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-11-19T08:25:33.717 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-11-19T08:25:33.728 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-11-19T08:25:33.916 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-11-19T08:25:33.927 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-11-19T08:25:34.115 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-11-19T08:25:34.126 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-11-19T08:25:34.315 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-11-19T08:25:34.338 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-11-19T08:25:34.507 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-11-19T08:25:34.540 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-11-19T08:25:34.708 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-11-19T08:25:34.751 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-11-19T08:25:34.908 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-11-19T08:25:34.960 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-11-19T08:25:35.106 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-11-19T08:25:35.183 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-11-19T08:25:35.301 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-11-19T08:25:35.386 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-11-19T08:25:35.502 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-11-19T08:25:35.599 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-11-19T08:25:35.706 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-11-19T08:25:35.800 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-11-19T08:25:35.905 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-11-19T08:25:36.011 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-11-19T08:25:36.107 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-11-19T08:25:36.223 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-11-19T08:25:36.296 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-11-19T08:25:36.424 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-11-19T08:25:36.497 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-11-19T08:25:36.634 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-11-19T08:25:36.697 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-11-19T08:25:36.844 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-11-19T08:25:36.898 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-11-19T08:25:37.045 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-11-19T08:25:37.100 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-11-19T08:25:37.252 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-11-19T08:25:37.295 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-11-19T08:25:37.454 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-11-19T08:25:37.498 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-19T08:25:37.659 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-11-19T08:25:37.692 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-11-19T08:25:37.870 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-19T08:25:37.891 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-11-19T08:25:38.080 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-11-19T08:25:38.091 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-11-19T08:25:38.280 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-11-19T08:25:38.292 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-11-19T08:25:38.481 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-11-19T08:25:38.493 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-11-19T08:25:38.672 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-11-19T08:25:38.705 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-11-19T08:25:38.874 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-11-19T08:25:38.908 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-11-19T08:25:39.076 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-19T08:25:39.119 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-11-19T08:25:39.279 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-11-19T08:25:39.333 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-11-19T08:25:39.482 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-11-19T08:25:39.535 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-19T08:25:39.673 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-11-19T08:25:39.746 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-11-19T08:25:39.862 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-11-19T08:25:39.947 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-11-19T08:25:40.073 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-11-19T08:25:40.161 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-11-19T08:25:40.269 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-11-19T08:25:40.376 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-11-19T08:25:40.472 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-11-19T08:25:40.579 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-11-19T08:25:40.675 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-11-19T08:25:40.782 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-11-19T08:25:40.870 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-11-19T08:25:41.000 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-11-19T08:25:41.075 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-11-19T08:25:41.216 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-11-19T08:25:41.271 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-11-19T08:25:41.416 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-11-19T08:25:41.469 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-11-19T08:25:41.626 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-11-19T08:25:41.659 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-11-19T08:25:41.825 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-11-19T08:25:41.857 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-11-19T08:25:42.034 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-11-19T08:25:42.055 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-11-19T08:25:42.241 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-11-19T08:25:42.253 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-11-19T08:25:42.440 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-19T08:25:42.441 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-11-19T08:25:42.640 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-11-19T08:25:42.651 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-11-19T08:25:42.828 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-11-19T08:25:42.850 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-11-19T08:25:43.035 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-11-19T08:25:43.057 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-11-19T08:25:43.240 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-11-19T08:25:43.274 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-11-19T08:25:43.430 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-11-19T08:25:43.483 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-11-19T08:25:43.631 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-11-19T08:25:43.695 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-11-19T08:25:43.831 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-11-19T08:25:43.894 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-11-19T08:25:44.031 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-11-19T08:25:44.105 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-11-19T08:25:44.231 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-11-19T08:25:44.316 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-11-19T08:25:44.431 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-11-19T08:25:44.526 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-11-19T08:25:44.631 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-11-19T08:25:44.725 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-11-19T08:25:44.829 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-11-19T08:25:44.934 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-11-19T08:25:45.028 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-11-19T08:25:45.143 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-11-19T08:25:45.218 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-11-19T08:25:45.350 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-11-19T08:25:45.423 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-11-19T08:25:45.565 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-11-19T08:25:45.629 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-11-19T08:25:45.774 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-11-19T08:25:45.817 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-11-19T08:25:45.974 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-11-19T08:25:46.016 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-11-19T08:25:46.184 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-11-19T08:25:46.217 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-11-19T08:25:46.385 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-11-19T08:25:46.406 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-11-19T08:25:46.606 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-11-19T08:25:46.608 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-11-19T08:25:46.799 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-11-19T08:25:46.812 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-11-19T08:25:47.003 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-11-19T08:25:47.014 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-11-19T08:25:47.205 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-11-19T08:25:47.228 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-11-19T08:25:47.397 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-11-19T08:25:47.423 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-11-19T08:25:47.593 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-11-19T08:25:47.648 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-11-19T08:25:47.800 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-11-19T08:25:47.852 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-11-19T08:25:47.988 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-11-19T08:25:48.064 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-11-19T08:25:48.191 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-11-19T08:25:48.266 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-11-19T08:25:48.392 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-11-19T08:25:48.476 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-11-19T08:25:48.582 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-11-19T08:25:48.678 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-11-19T08:25:48.785 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-11-19T08:25:48.890 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-11-19T08:25:48.974 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-11-19T08:25:49.090 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-11-19T08:25:49.174 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-11-19T08:25:49.298 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-11-19T08:25:49.371 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-11-19T08:25:49.509 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-11-19T08:25:49.562 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-19T08:25:49.709 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-11-19T08:25:49.763 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-11-19T08:25:49.918 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-11-19T08:25:49.961 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-11-19T08:25:50.120 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-11-19T08:25:50.153 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-11-19T08:25:50.324 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-11-19T08:25:50.346 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-11-19T08:25:50.547 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-11-19T08:25:50.549 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-11-19T08:25:50.737 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-11-19T08:25:50.758 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-11-19T08:25:50.947 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-11-19T08:25:50.948 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-11-19T08:25:51.150 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-11-19T08:25:51.350 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-11-19T08:25:51.374 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-11-19T08:25:51.543 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-11-19T08:25:51.576 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-11-19T08:25:51.743 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-11-19T08:25:51.786 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-11-19T08:25:51.942 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-11-19T08:25:51.996 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-11-19T08:25:52.133 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-11-19T08:25:52.200 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-11-19T08:25:52.202 INFO:teuthology.task.ansible.out: 2021-11-19T08:25:52.328 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-11-19T08:25:52.411 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-11-19T08:25:52.526 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-11-19T08:25:52.611 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-11-19T08:25:52.727 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-11-19T08:25:52.822 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-11-19T08:25:52.918 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-11-19T08:25:53.025 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-11-19T08:25:53.121 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-11-19T08:25:53.237 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-11-19T08:25:53.323 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-11-19T08:25:53.440 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-11-19T08:25:53.518 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-11-19T08:25:53.645 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-11-19T08:25:53.720 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-11-19T08:25:53.858 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-11-19T08:25:53.923 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-11-19T08:25:54.061 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-11-19T08:25:54.115 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-11-19T08:25:54.273 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-11-19T08:25:54.316 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-11-19T08:25:54.474 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-11-19T08:25:54.508 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-11-19T08:25:54.679 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-11-19T08:25:54.711 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-11-19T08:25:54.890 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-11-19T08:25:54.902 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-11-19T08:25:55.091 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-11-19T08:25:55.102 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-11-19T08:25:55.293 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-11-19T08:25:55.305 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-11-19T08:25:55.494 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-11-19T08:25:55.518 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-11-19T08:25:55.694 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-11-19T08:25:55.716 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-11-19T08:25:55.896 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-11-19T08:25:55.938 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-11-19T08:25:56.094 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-11-19T08:25:56.147 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-11-19T08:25:56.295 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-11-19T08:25:56.347 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-11-19T08:25:56.494 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-11-19T08:25:56.557 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-11-19T08:25:56.694 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-11-19T08:25:56.762 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-11-19T08:25:56.883 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-11-19T08:25:56.966 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-11-19T08:25:57.174 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-11-19T08:25:57.377 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-11-19T08:25:57.586 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-11-19T08:25:57.785 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-11-19T08:25:57.994 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-11-19T08:25:58.194 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-11-19T08:25:58.445 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-11-19T08:25:59.932 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-11-19T08:26:00.350 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-11-19T08:26:01.712 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-11-19T08:26:01.828 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-11-19T08:26:03.319 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-11-19T08:26:03.341 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-11-19T08:26:04.861 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-11-19T08:26:05.244 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-11-19T08:26:06.394 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-11-19T08:26:06.992 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-11-19T08:26:07.960 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-11-19T08:26:08.640 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-11-19T08:26:09.538 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-11-19T08:26:10.277 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-11-19T08:26:11.140 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-11-19T08:26:11.925 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-11-19T08:26:12.797 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-11-19T08:26:13.670 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-11-19T08:26:14.391 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-11-19T08:26:15.602 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-11-19T08:26:16.115 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-11-19T08:26:17.604 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-11-19T08:26:17.771 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-11-19T08:26:19.314 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-11-19T08:26:19.492 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-11-19T08:26:21.126 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-11-19T08:26:21.211 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-11-19T08:26:22.827 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-11-19T08:26:22.902 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-11-19T08:26:24.582 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-11-19T08:26:24.594 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-11-19T08:26:26.243 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-11-19T08:26:26.390 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-11-19T08:26:27.920 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-11-19T08:26:28.151 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-11-19T08:26:29.584 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-11-19T08:26:29.895 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-11-19T08:26:31.242 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10122, "groups": "sudo", "home": "/home/choffman", "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "name": "choffman", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10122} 2021-11-19T08:26:31.389 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:26:31 +0000 (0:01:00.851) 0:01:08.248 ******* 2021-11-19T08:26:31.459 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-11-19T08:26:31.481 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:26:31 +0000 (0:00:00.091) 0:01:08.340 ******* 2021-11-19T08:26:31.583 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-11-19T08:26:31.585 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@desktop 2021-11-19T08:26:31.587 INFO:teuthology.task.ansible.out: QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2021-11-19T08:26:31.589 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed20 2021-11-19T08:26:31.591 INFO:teuthology.task.ansible.out:82caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name" 2021-11-19T08:26:31.592 INFO:teuthology.task.ansible.out:: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915 2021-11-19T08:26:31.594 INFO:teuthology.task.ansible.out:d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZ 2021-11-19T08:26:31.596 INFO:teuthology.task.ansible.out:YM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 2021-11-19T08:26:31.597 INFO:teuthology.task.ansible.out: 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp 2021-11-19T08:26:31.607 INFO:teuthology.task.ansible.out:", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3 2021-11-19T08:26:31.609 INFO:teuthology.task.ansible.out:624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9Jj 2021-11-19T08:26:31.610 INFO:teuthology.task.ansible.out:enDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "sh 2021-11-19T08:26:31.612 INFO:teuthology.task.ansible.out:yamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a 2021-11-19T08:26:31.614 INFO:teuthology.task.ansible.out:28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5 2021-11-19T08:26:31.615 INFO:teuthology.task.ansible.out:299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": " 2021-11-19T08:26:31.617 INFO:teuthology.task.ansible.out:zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}]}, "changed": false} 2021-11-19T08:26:31.645 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:26:31 +0000 (0:00:00.163) 0:01:08.503 ******* 2021-11-19T08:26:31.683 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-11-19T08:26:31.684 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-11-19T08:26:32.589 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-11-19T08:26:32.591 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"after": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "attempts": 1, "before": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "changed": false, "remote_url_changed": false} 2021-11-19T08:26:32.611 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:26:32 +0000 (0:00:00.966) 0:01:09.470 ******* 2021-11-19T08:26:32.952 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-19T08:26:32.954 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-11-19T08:26:33.161 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-11-19T08:26:33.369 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-11-19T08:26:33.454 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-11-19T08:26:33.456 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-11-19T08:26:33.584 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw 2021-11-19T08:26:33.586 INFO:teuthology.task.ansible.out:8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-11-19T08:26:33.789 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-11-19T08:26:33.998 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-11-19T08:26:34.000 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-11-19T08:26:34.206 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-11-19T08:26:34.414 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-11-19T08:26:34.624 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-11-19T08:26:34.636 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:26:34.846 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-11-19T08:26:34.847 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-11-19T08:26:35.056 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-11-19T08:26:35.058 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-11-19T08:26:35.068 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:26:35.079 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:26:35.257 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-11-19T08:26:35.258 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10122, "groups": "sudo", "home": "/home/choffman", "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "name": "choffman", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10122} 2021-11-19T08:26:35.375 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-19T08:26:35.377 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-11-19T08:26:35.398 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:26:35 +0000 (0:00:02.786) 0:01:12.256 ******* 2021-11-19T08:26:35.504 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-11-19T08:26:35.507 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-11-19T08:26:35.589 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-11-19T08:26:35.613 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:26:35 +0000 (0:00:00.215) 0:01:12.472 ******* 2021-11-19T08:26:35.704 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-11-19T08:26:35.706 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@desktop 2021-11-19T08:26:35.708 INFO:teuthology.task.ansible.out: QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2021-11-19T08:26:35.709 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed20 2021-11-19T08:26:35.711 INFO:teuthology.task.ansible.out:82caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name" 2021-11-19T08:26:35.713 INFO:teuthology.task.ansible.out:: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9eb 2021-11-19T08:26:35.713 INFO:teuthology.task.ansible.out:ba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915 2021-11-19T08:26:35.719 INFO:teuthology.task.ansible.out:d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZ 2021-11-19T08:26:35.720 INFO:teuthology.task.ansible.out:YM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 2021-11-19T08:26:35.722 INFO:teuthology.task.ansible.out: 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp 2021-11-19T08:26:35.723 INFO:teuthology.task.ansible.out:", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3 2021-11-19T08:26:35.725 INFO:teuthology.task.ansible.out:624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9Jj 2021-11-19T08:26:35.726 INFO:teuthology.task.ansible.out:enDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "sh 2021-11-19T08:26:35.728 INFO:teuthology.task.ansible.out:yamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a 2021-11-19T08:26:35.730 INFO:teuthology.task.ansible.out:28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5 2021-11-19T08:26:35.730 INFO:teuthology.task.ansible.out:299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": " 2021-11-19T08:26:35.731 INFO:teuthology.task.ansible.out:zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}]}, "changed": false} 2021-11-19T08:26:35.733 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-19T08:26:35.735 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-11-19T08:26:35.760 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:26:35 +0000 (0:00:00.147) 0:01:12.619 ******* 2021-11-19T08:26:35.924 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-11-19T08:26:36.134 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-11-19T08:26:36.343 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-11-19T08:26:36.345 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-11-19T08:26:36.420 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-11-19T08:26:36.421 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"after": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "attempts": 1, "before": "d72e06d3c78c8d9639a8ca75effa043ee468910f", "changed": false, "remote_url_changed": false} 2021-11-19T08:26:36.442 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:26:36 +0000 (0:00:00.681) 0:01:13.300 ******* 2021-11-19T08:26:36.554 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-19T08:26:36.555 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-11-19T08:26:36.557 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-11-19T08:26:36.766 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-11-19T08:26:36.916 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-11-19T08:26:36.970 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-11-19T08:26:37.137 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-11-19T08:26:37.181 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-11-19T08:26:37.183 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-11-19T08:26:37.360 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-11-19T08:26:37.382 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-11-19T08:26:37.583 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-11-19T08:26:37.585 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-11-19T08:26:37.594 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-11-19T08:26:37.803 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-11-19T08:26:37.805 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-11-19T08:26:38.022 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-11-19T08:26:38.024 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-11-19T08:26:38.231 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-11-19T08:26:38.242 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-11-19T08:26:38.440 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-11-19T08:26:38.442 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-11-19T08:26:38.451 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-11-19T08:26:38.640 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-11-19T08:26:38.673 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-11-19T08:26:38.685 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:26:38.853 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-11-19T08:26:38.896 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-11-19T08:26:38.898 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-11-19T08:26:39.066 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-11-19T08:26:39.129 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-11-19T08:26:39.131 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-11-19T08:26:39.141 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:26:39.152 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:26:39.268 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-11-19T08:26:39.383 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-11-19T08:26:39.478 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-19T08:26:39.594 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-11-19T08:26:39.596 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-11-19T08:26:39.690 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-11-19T08:26:39.817 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-11-19T08:26:39.903 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-11-19T08:26:40.030 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-11-19T08:26:40.116 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-11-19T08:26:40.118 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-11-19T08:26:40.255 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-11-19T08:26:40.319 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-11-19T08:26:40.321 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-11-19T08:26:40.484 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-11-19T08:26:40.486 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-11-19T08:26:40.517 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-11-19T08:26:40.710 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-11-19T08:26:40.712 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-11-19T08:26:40.732 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-11-19T08:26:40.932 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-11-19T08:26:40.954 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-11-19T08:26:41.146 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-11-19T08:26:41.157 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-11-19T08:26:41.158 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-19T08:26:41.367 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-11-19T08:26:41.369 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-11-19T08:26:41.370 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-11-19T08:26:41.378 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-11-19T08:26:41.380 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-11-19T08:26:41.578 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-11-19T08:26:41.588 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-11-19T08:26:41.778 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-11-19T08:26:41.811 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-11-19T08:26:41.990 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-11-19T08:26:42.022 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-11-19T08:26:42.199 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-11-19T08:26:42.243 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-11-19T08:26:42.410 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-11-19T08:26:42.464 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-11-19T08:26:42.631 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-11-19T08:26:42.632 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-11-19T08:26:42.690 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-11-19T08:26:42.693 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-11-19T08:26:42.829 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-11-19T08:26:42.905 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-11-19T08:26:43.042 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-11-19T08:26:43.126 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-11-19T08:26:43.251 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-11-19T08:26:43.253 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-11-19T08:26:43.346 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-11-19T08:26:43.453 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-11-19T08:26:43.579 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-11-19T08:26:43.664 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-11-19T08:26:43.791 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-19T08:26:43.875 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-11-19T08:26:43.877 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-11-19T08:26:44.013 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-11-19T08:26:44.087 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-11-19T08:26:44.244 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-11-19T08:26:44.298 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-11-19T08:26:44.455 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-11-19T08:26:44.457 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-11-19T08:26:44.509 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-11-19T08:26:44.678 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-11-19T08:26:44.680 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-11-19T08:26:44.711 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-19T08:26:44.900 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-11-19T08:26:44.923 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-11-19T08:26:45.122 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-11-19T08:26:45.133 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-11-19T08:26:45.333 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-11-19T08:26:45.344 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-11-19T08:26:45.547 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-11-19T08:26:45.551 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-19T08:26:45.577 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-11-19T08:26:45.771 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-11-19T08:26:45.774 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-11-19T08:26:45.777 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-11-19T08:26:45.793 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-11-19T08:26:45.997 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD 2021-11-19T08:26:45.998 INFO:teuthology.task.ansible.out:6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-11-19T08:26:46.208 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-11-19T08:26:46.209 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-11-19T08:26:46.420 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-11-19T08:26:46.431 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-11-19T08:26:46.622 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-11-19T08:26:46.656 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-11-19T08:26:46.824 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-11-19T08:26:46.878 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-11-19T08:26:47.035 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-11-19T08:26:47.099 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-11-19T08:26:47.101 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-11-19T08:26:47.246 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-11-19T08:26:47.310 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-11-19T08:26:47.456 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-11-19T08:26:47.519 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-11-19T08:26:47.657 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-11-19T08:26:47.752 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-11-19T08:26:47.754 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-11-19T08:26:47.868 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-11-19T08:26:47.972 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-11-19T08:26:48.077 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-11-19T08:26:48.193 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-11-19T08:26:48.298 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-11-19T08:26:48.300 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-11-19T08:26:48.413 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-11-19T08:26:48.415 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-11-19T08:26:48.508 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-11-19T08:26:48.624 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-11-19T08:26:48.718 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-11-19T08:26:48.850 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-11-19T08:26:48.918 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-11-19T08:26:49.056 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-11-19T08:26:49.132 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-11-19T08:26:49.134 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-11-19T08:26:49.280 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-11-19T08:26:49.344 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-11-19T08:26:49.492 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-11-19T08:26:49.556 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-11-19T08:26:49.703 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-11-19T08:26:49.768 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-11-19T08:26:49.918 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-11-19T08:26:49.971 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-11-19T08:26:50.138 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-11-19T08:26:50.173 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-11-19T08:26:50.352 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-11-19T08:26:50.387 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-11-19T08:26:50.571 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-11-19T08:26:50.593 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-11-19T08:26:50.784 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-11-19T08:26:50.806 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-11-19T08:26:50.808 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-11-19T08:26:51.004 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-11-19T08:26:51.015 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-11-19T08:26:51.226 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-11-19T08:26:51.227 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-11-19T08:26:51.426 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-11-19T08:26:51.448 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-11-19T08:26:51.637 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-11-19T08:26:51.669 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-11-19T08:26:51.850 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-11-19T08:26:51.981 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-11-19T08:26:52.098 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-11-19T08:26:52.120 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-11-19T08:26:52.310 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-11-19T08:26:52.312 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-19T08:26:52.353 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-11-19T08:26:52.519 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-11-19T08:26:52.574 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-11-19T08:26:52.743 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-11-19T08:26:52.796 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-11-19T08:26:52.964 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-11-19T08:26:53.027 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-11-19T08:26:53.029 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-11-19T08:26:53.172 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-11-19T08:26:53.247 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-11-19T08:26:53.377 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-11-19T08:26:53.475 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-11-19T08:26:53.581 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-11-19T08:26:53.686 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-11-19T08:26:53.791 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-11-19T08:26:53.913 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-11-19T08:26:53.915 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-11-19T08:26:54.007 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-11-19T08:26:54.133 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-11-19T08:26:54.217 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-11-19T08:26:54.354 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-11-19T08:26:54.418 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-11-19T08:26:54.566 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-11-19T08:26:54.631 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-11-19T08:26:54.776 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-11-19T08:26:54.830 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-11-19T08:26:54.998 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-11-19T08:26:55.051 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-11-19T08:26:55.219 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-11-19T08:26:55.251 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-11-19T08:26:55.434 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-11-19T08:26:55.468 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-11-19T08:26:55.658 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-11-19T08:26:55.660 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-11-19T08:26:55.669 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-11-19T08:26:55.879 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-11-19T08:26:55.881 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-11-19T08:26:56.105 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devB 2021-11-19T08:26:56.108 INFO:teuthology.task.ansible.out:ox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-11-19T08:26:56.314 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-11-19T08:26:56.317 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-11-19T08:26:56.319 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-11-19T08:26:56.536 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-11-19T08:26:56.537 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-11-19T08:26:56.746 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-11-19T08:26:56.757 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-11-19T08:26:56.956 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-11-19T08:26:56.968 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-11-19T08:26:57.201 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-11-19T08:26:57.214 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-11-19T08:26:57.216 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-11-19T08:26:57.406 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-11-19T08:26:57.439 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-11-19T08:26:57.608 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-11-19T08:26:57.661 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-11-19T08:26:57.824 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-11-19T08:26:57.877 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-11-19T08:26:58.034 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-11-19T08:26:58.099 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-11-19T08:26:58.235 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-11-19T08:26:58.310 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-11-19T08:26:58.447 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-11-19T08:26:58.530 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-11-19T08:26:58.656 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-11-19T08:26:58.658 INFO:teuthology.task.ansible.out: 2021-11-19T08:26:58.752 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-11-19T08:26:58.857 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-11-19T08:26:58.963 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-11-19T08:26:59.067 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4m0LAcC7XiFiXFtYcCvZqX1CtyV9JU+VUEawZJpWrGTOr8c+vIexq7+WKg+eIUMot4+dtfwG8lByS1VofglIe5/quv17Xj7dO/hF2WkV8O7RYt7NhzmrKg54yB73pbRpvqzGceWWDvGRsWQGzbY7ARqOp4FPIDJvgXABCkak+hmjXyXpvXN418CBDbDmxHWwFb9QtB/MpQFiFrfl1EMDj8DaXZNWgo5J/qytic6LgiyPHhHfhSylk4cl9TGftNatobpMjqBT0O2/plk/qf+7pHPuvGSgihjijdpEPzVfctXE2OmsQAIZgkzXeCqts+s6ZYdUKq1FvpV5rMM8ZLqfCkDjEzFENJS68YYE2EjxAEgMzj7h/sWc4B5I5pxkAqVAQSGjYRu6rCCbFz8PXUdTyW+Vs1YeylnlBMiCa4xE4gwvW3HlWEnumtyURd1PRENrGJsJ/0uX6bpltbSg4sGACRYpLPJMm3OhqI3Jm/J/e2NLjfCw+svU2447xnfJOh68= pdhange@vaquita\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoeyVaMjhcGrWQB+4Rq+SSIi8+OY/kx2s1vofiGt1gu3RNtFmG0Df1VITJJ32dbzaW7kTNxaRvhldNd/n7yEMssG+Wg+a5WKvSudlbVMmscbSXdFuVbcCW1jNrTGCYrxL7Sn5e7NQEQfAsCbfXwTcYBw33NdBVovC+gs2V3sZGvX3Y0akZONNuVg1mhu0L4m1/5IegJh005KeilwetHA2A5Dr10RG6oyIgT7t4z5gZXsrOiG3UQBXaekILYV9jxMWoFXGkVLc8TpL9SUAdxE3Owc0u1XiPqYYlEN+zDvE0554Zj8Oc2ZO6HRXCZXQFvzXeCD12YswcOfkIDmJRF7GT prashant@mauli", "key_options": null, "k 2021-11-19T08:26:59.069 INFO:teuthology.task.ansible.out:eyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-11-19T08:26:59.182 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-11-19T08:26:59.297 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-11-19T08:26:59.403 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-11-19T08:26:59.510 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-11-19T08:26:59.626 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-11-19T08:26:59.711 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-11-19T08:26:59.713 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-11-19T08:26:59.859 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-11-19T08:26:59.933 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCt4sQTrDDg562IWY9Bw2fxFR7Y7b5lfo9egHa1qNFR5K4gt8lUwQXgzcxZK5dgSTZXHDGxrbYKWsdb8NbpgU/YjSqDUIptMZrhTI6pdHKZ2usY+nieaJj0NIZP/S7M93BbJeoNBYACjvDUr+vXEg6ZL3uAj+zFXr/4te+p4NqJBl4bSpvjcRK12O+F6HE3Q93j0Vlfs8IbXP4vcPcqOW+2usXw7JPdfjwuACJxrLTx/0mBsv8qXJDfOpPIHAMA/9SdTAzhs6XMo+fOwVOKkV+ki2fO3sl8pE16aFOkI3cYX36nkYYpZ2bJLnIPcLv37i3Nypl8eNLA4gzjkDf1o/KLO5vXOs76H189WGLqjh0SHAx/3KO6mrIhTq8Bv6YWoT3ntg22fXaxeajNhHcQ0bKZtGOAb0W1FcMRbSTkgEFAassYxlpTtuaCDIuA4fUVKW97Zu62eDDsSUtTWak/Yf940qAbptPPAw4ljmrvZRcjM9y6mGU8lKi5H4QoEOfbPok= nia@fedora", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-11-19T08:27:00.072 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-11-19T08:27:00.146 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-11-19T08:27:00.284 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-11-19T08:27:00.358 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-11-19T08:27:00.361 INFO:teuthology.task.ansible.out: 2021-11-19T08:27:00.502 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-11-19T08:27:00.566 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-11-19T08:27:00.715 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-11-19T08:27:00.769 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-11-19T08:27:00.937 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-11-19T08:27:00.980 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-11-19T08:27:01.158 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-11-19T08:27:01.190 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-11-19T08:27:01.370 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-11-19T08:27:01.372 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-11-19T08:27:01.402 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-11-19T08:27:01.593 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-11-19T08:27:01.625 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-11-19T08:27:01.803 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-11-19T08:27:01.842 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-11-19T08:27:02.022 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-11-19T08:27:02.115 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-11-19T08:27:02.241 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-11-19T08:27:02.326 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-11-19T08:27:02.464 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-11-19T08:27:02.528 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-11-19T08:27:02.673 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-11-19T08:27:02.738 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-11-19T08:27:02.895 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-11-19T08:27:02.959 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-11-19T08:27:03.108 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-11-19T08:27:03.171 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-11-19T08:27:03.328 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-11-19T08:27:03.381 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-11-19T08:27:03.538 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-11-19T08:27:03.591 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-11-19T08:27:03.762 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-11-19T08:27:03.795 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-11-19T08:27:03.984 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-11-19T08:27:04.007 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-11-19T08:27:04.197 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4m0LAcC7XiFiXFtYcCvZqX1CtyV9JU+VUEawZJpWrGTOr8c+vIexq7+WKg+eIUMot4+dtfwG8lByS1VofglIe5/quv17Xj7dO/hF2WkV8O7RYt7NhzmrKg54yB73pbRpvqzGceWWDvGRsWQGzbY7ARqOp4FPIDJvgXABCkak+hmjXyXpvXN418CBDbDmxHWwFb9QtB/MpQFiFrfl1EMDj8DaXZNWgo5J/qytic6LgiyPHhHfhSylk4cl9TGftNatobpMjqBT0O2/plk/qf+7pHPuvGSgihjijdpEPzVfctXE2OmsQAIZgkzXeCqts+s6ZYdUKq1FvpV5rMM8ZLqfCkDjEzFENJS68YYE2EjxAEgMzj7h/sWc4B5I5pxkAqVAQSGjYRu6rCCbFz8PXUdTyW+Vs1YeylnlBMiCa4xE4gwvW3HlWEnumtyURd1PRENrGJsJ/0uX6bpltbSg4sGACRYpLPJMm3OhqI3Jm/J/e2NLjfCw+svU2447xnfJOh68= pdhange@vaquita\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoeyVaMjhcGrWQB+4Rq+SSIi8+OY/kx2s1vofiGt1gu3RNtFmG0Df1VITJJ32dbzaW7kTNxaRvhldNd/n7yEMssG+Wg+a5WKvSudlbVMmscbSXdFuVbcCW1jNrTGCYrxL7Sn5e7NQEQfAsCbfXwTcYBw33NdBVovC+gs2V3sZGvX3Y0akZONNuVg1mhu0L4m1/5IegJh005KeilwetHA2A5Dr10RG6oyIgT7t4z5gZXsrOiG3UQBXaekILYV9jxMWoFXGkVLc8TpL9SUAdxE3Owc0u1XiPqYYlEN+zDvE0554Zj8Oc2ZO6HRXCZXQFvzXeCD12YswcOfkIDmJRF7GT prashant@mauli", "key_options": null, "k 2021-11-19T08:27:04.198 INFO:teuthology.task.ansible.out:eyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-11-19T08:27:04.219 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-11-19T08:27:04.419 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-11-19T08:27:04.430 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-11-19T08:27:04.651 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7LPHiaPQXdE+yt9cn7UMuRr8k/LjSUmMrnrAqp8ySQtX6OA3/NcOsxg6FJ45chZqeoXwhsYEW+7pTInDqFlOlmVynR26lWnZfk9DjBJZcloUCn8y5MR9QitNYsOMs5KwCEFmBH6sP+2vCFocXBnKQcVZLRfip3MEG4LoWYfI4ir7fLmK1Hr5G+GhtMstXvVSG4VE4IgbU9/LRIOv7P9qFn4bRJoUSiS2dDbUxVIuneA9bIEnB9ZEVGahEzlUhx6qDSs0FvrnneCQ1ayOFpxClzjT5GuKk/J3LJvWAw8CAwDXCf3FzsWQsvb5ru5VSazUI9uA3o9wWkbLW2oZPgtzI8lmwbL+MT58T92tqaJOPID3IZDGjzNrRqU5BQYpvFYHpHRRiqV625og+ktYfXgO+dTbgAxScomqutydI4cyJusDtnWtljWnEocS+2hUSZpmB6B3Wj70TuuzsbYrcrJ/NgRpQNX4dwHCJl6vPT+xkoLh1imYAC7klf7cnrOyGcs= choffman@choffman.net", "key_options": null, "keyfile": "/home/choffman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "choffman", "validate_certs": true} 2021-11-19T08:27:04.653 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-11-19T08:27:04.877 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:04 +0000 (0:00:28.435) 0:01:41.736 ******* 2021-11-19T08:27:04.884 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-11-19T08:27:04.886 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-11-19T08:27:04.962 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-19T08:27:04.964 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:04.974 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:04.985 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:04.986 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:04.996 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:04.998 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.008 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.008 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.082 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-19T08:27:05.084 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCt4sQTrDDg562IWY9Bw2fxFR7Y7b5lfo9egHa1qNFR5K4gt8lUwQXgzcxZK5dgSTZXHDGxrbYKWsdb8NbpgU/YjSqDUIptMZrhTI6pdHKZ2usY+nieaJj0NIZP/S7M93BbJeoNBYACjvDUr+vXEg6ZL3uAj+zFXr/4te+p4NqJBl4bSpvjcRK12O+F6HE3Q93j0Vlfs8IbXP4vcPcqOW+2usXw7JPdfjwuACJxrLTx/0mBsv8qXJDfOpPIHAMA/9SdTAzhs6XMo+fOwVOKkV+ki2fO3sl8pE16aFOkI3cYX36nkYYpZ2bJLnIPcLv37i3Nypl8eNLA4gzjkDf1o/KLO5vXOs76H189WGLqjh0SHAx/3KO6mrIhTq8Bv6YWoT3ntg22fXaxeajNhHcQ0bKZtGOAb0W1FcMRbSTkgEFAassYxlpTtuaCDIuA4fUVKW97Zu62eDDsSUtTWak/Yf940qAbptPPAw4ljmrvZRcjM9y6mGU8lKi5H4QoEOfbPok= nia@fedora", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-11-19T08:27:05.230 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-19T08:27:05.232 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e 2021-11-19T08:27:05.234 INFO:teuthology.task.ansible.out:1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.242 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.305 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-19T08:27:05.307 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-11-19T08:27:05.462 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-11-19T08:27:05.527 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-11-19T08:27:05.659 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-19T08:27:05.660 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-11-19T08:27:05.670 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.681 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.684 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.693 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.704 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.706 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.715 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.726 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.728 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.738 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.740 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.751 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.762 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.763 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.774 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.776 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.786 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.788 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.797 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.808 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.819 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.821 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.831 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.843 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.845 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.854 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.856 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.866 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.877 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.889 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.900 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.902 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.911 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.923 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.925 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.934 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.936 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.956 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.958 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.968 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.970 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.980 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.980 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-19T08:27:05.981 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-11-19T08:27:05.992 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:05.994 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.003 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.005 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.015 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.025 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.027 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.037 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.038 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.048 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.059 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.061 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.071 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.072 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.083 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.094 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.096 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.105 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.107 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.117 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.128 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.129 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.139 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.141 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.151 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.162 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.164 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.173 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.176 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.186 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.198 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.200 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-11-19T08:27:06.210 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.212 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.221 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.233 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.235 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.244 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.246 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.256 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.267 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.268 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.279 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.289 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.291 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.300 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.302 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.312 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.323 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.325 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.335 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.337 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.347 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.358 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.369 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.380 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.391 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.393 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.403 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.415 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.417 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.426 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-11-19T08:27:06.428 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.438 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.440 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.450 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.461 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.472 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.484 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.495 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.497 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.506 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.517 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.519 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.529 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.531 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.541 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.553 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.563 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.565 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.575 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.587 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.589 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.598 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.609 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.612 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.621 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.632 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.634 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.645 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.659 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-11-19T08:27:06.662 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.673 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.674 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.726 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:06 +0000 (0:00:01.848) 0:01:43.585 ******* 2021-11-19T08:27:06.797 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:06.820 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:06 +0000 (0:00:00.093) 0:01:43.678 ******* 2021-11-19T08:27:06.860 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-11-19T08:27:07.094 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-11-19T08:27:07.106 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi134.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-11-19T08:27:07.308 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-11-19T08:27:07.331 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-11-19T08:27:07.510 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:07.512 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-11-19T08:27:07.560 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-11-19T08:27:07.706 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:07.709 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-11-19T08:27:07.784 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-19T08:27:07.787 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-11-19T08:27:07.903 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi134.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-11-19T08:27:08.001 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-11-19T08:27:08.096 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:08.099 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-11-19T08:27:08.223 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-19T08:27:08.225 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-11-19T08:27:08.286 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:08.289 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-11-19T08:27:08.435 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-11-19T08:27:08.478 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi134.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-11-19T08:27:08.658 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-11-19T08:27:08.690 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi134.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-11-19T08:27:08.883 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-11-19T08:27:08.884 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-11-19T08:27:09.074 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:09.076 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-11-19T08:27:09.106 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-19T08:27:09.107 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-11-19T08:27:09.274 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi134.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-11-19T08:27:09.326 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-19T08:27:09.328 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-11-19T08:27:09.462 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:09.464 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-11-19T08:27:09.536 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-11-19T08:27:09.538 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-11-19T08:27:09.653 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi134.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-11-19T08:27:09.758 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-11-19T08:27:09.854 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:09.856 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-11-19T08:27:09.983 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi023.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM7LPHiaPQXdE+yt9cn7UMuRr8k/LjSUmMrnrAqp8ySQtX6OA3/NcOsxg6FJ45chZqeoXwhsYEW+7pTInDqFlOlmVynR26lWnZfk9DjBJZcloUCn8y5MR9QitNYsOMs5KwCEFmBH6sP+2vCFocXBnKQcVZLRfip3MEG4LoWYfI4ir7fLmK1Hr5G+GhtMstXvVSG4VE4IgbU9/LRIOv7P9qFn4bRJoUSiS2dDbUxVIuneA9bIEnB9ZEVGahEzlUhx6qDSs0FvrnneCQ1ayOFpxClzjT5GuKk/J3LJvWAw8CAwDXCf3FzsWQsvb5ru5VSazUI9uA3o9wWkbLW2oZPgtzI8lmwbL+MT58T92tqaJOPID3IZDGjzNrRqU5BQYpvFYHpHRRiqV625og+ktYfXgO+dTbgAxScomqutydI4cyJusDtnWtljWnEocS+2hUSZpmB6B3Wj70TuuzsbYrcrJ/NgRpQNX4dwHCJl6vPT+xkoLh1imYAC7klf7cnrOyGcs= choffman@choffman.net", "key_options": null, "keyfile": "/home/choffman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "choffman", "validate_certs": true} 2021-11-19T08:27:10.069 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:10.071 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-11-19T08:27:10.092 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:10 +0000 (0:00:03.271) 0:01:46.950 ******* 2021-11-19T08:27:10.179 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-19T08:27:10.180 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.191 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.192 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.203 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.214 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.216 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.229 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.231 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@desktop QI358a0dizGe6T42hCYj1A 139ab306be981d0a669ec2288a2529503882deffceb28bdfc9bde2b70a825898"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.241 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.252 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:10.253 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-11-19T08:27:10.453 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-11-19T08:27:10.464 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-19T08:27:10.466 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-11-19T08:27:10.486 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.497 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.655 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:10.657 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-11-19T08:27:10.719 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-19T08:27:10.721 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-11-19T08:27:10.846 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:10.848 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-11-19T08:27:10.931 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-19T08:27:10.933 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-11-19T08:27:10.943 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.955 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.956 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.966 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.977 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.979 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:10.989 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.000 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.002 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.012 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.023 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.034 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.035 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.045 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.047 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi134.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.058 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.059 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.070 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.082 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.083 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.094 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.096 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.106 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.118 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.122 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.130 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.132 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.142 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.154 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.156 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.165 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.167 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.177 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.189 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.190 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.201 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.202 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@snowmon CjHOevVvBm+PwhjnhZYM7g 7393d3a95b1874764fc4e7ae2d481f1859b336e16e1ba53c49d126585e11ba56"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.212 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.223 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.225 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.235 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:11.237 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.247 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.249 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.259 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.271 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.282 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.293 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.295 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.305 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.316 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.318 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.327 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.329 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.339 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.352 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.353 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.363 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.365 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.374 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.387 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.389 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.399 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.401 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.411 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.424 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.426 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.447 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:11.448 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.449 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.449 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsal 2021-11-19T08:27:11.451 INFO:teuthology.task.ansible.out:omon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.461 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.472 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.474 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.483 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.485 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.494 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.506 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.508 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.517 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.519 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.528 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.539 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.541 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.550 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.562 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.572 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.574 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.583 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.594 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.596 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.606 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.617 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.628 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.630 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi134.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.640 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.642 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.651 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.662 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.664 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.674 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.675 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.685 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.696 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.698 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.707 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.709 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.719 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.721 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.730 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.741 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.743 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.753 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "pdhange@mauli hqKEk1OuI2Wmi1CAcJk5CQ f49d854572b5efe9514b4d741e336adb3534d00b22b0a505142d82eccc2f7535"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.754 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.764 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.775 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat OA+N80Ps0Ua7PfuVF4+M8w a9b0ecf7a28ef91e0653b0193f3f95967d6a671eff21ee8a99620a5a972ba409"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.786 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.788 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.798 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.809 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.821 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.822 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi134.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-11-19T08:27:11.822 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.833 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.835 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.844 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.856 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.857 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.867 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.869 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.879 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.890 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.892 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.901 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.903 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.914 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.926 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'choffman', 'ovpn': 'choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "choffman", "ovpn": "choffman@choffman.net 56rg/lHPXPRyVivND7+0Uw ca591258cda719a5a0bf5d5f4fa87976773625249d29ff6012ad6efeac33da2c"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:11.978 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:11 +0000 (0:00:01.886) 0:01:48.837 ******* 2021-11-19T08:27:12.016 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:12.018 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-11-19T08:27:12.060 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-11-19T08:27:12.062 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:12.084 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:12 +0000 (0:00:00.105) 0:01:48.942 ******* 2021-11-19T08:27:12.208 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:12.210 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-11-19T08:27:12.378 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-11-19T08:27:12.410 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-11-19T08:27:12.589 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-11-19T08:27:12.600 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-11-19T08:27:12.779 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-11-19T08:27:12.790 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-11-19T08:27:12.980 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-11-19T08:27:12.982 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-11-19T08:27:13.194 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} ok: [smithi023.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-11-19T08:27:13.384 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-11-19T08:27:13.395 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-11-19T08:27:13.573 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-11-19T08:27:13.595 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-11-19T08:27:13.774 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-11-19T08:27:13.796 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-11-19T08:27:13.962 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-11-19T08:27:14.004 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-11-19T08:27:14.151 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-11-19T08:27:14.205 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-11-19T08:27:14.342 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-11-19T08:27:14.406 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-11-19T08:27:14.535 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-11-19T08:27:14.610 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-11-19T08:27:14.819 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-11-19T08:27:15.019 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-11-19T08:27:15.219 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-11-19T08:27:15.419 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-11-19T08:27:15.620 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-11-19T08:27:15.819 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-11-19T08:27:16.027 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-11-19T08:27:16.102 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-11-19T08:27:16.227 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-11-19T08:27:16.435 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-11-19T08:27:16.634 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-11-19T08:27:16.846 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-11-19T08:27:17.045 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-11-19T08:27:17.244 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-11-19T08:27:17.443 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-11-19T08:27:17.654 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-11-19T08:27:17.751 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-11-19T08:27:17.860 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-11-19T08:27:18.062 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-11-19T08:27:18.275 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-11-19T08:27:18.476 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-11-19T08:27:18.681 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-11-19T08:27:18.882 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-11-19T08:27:19.087 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-11-19T08:27:19.284 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-11-19T08:27:19.404 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-11-19T08:27:19.488 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-11-19T08:27:19.687 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-11-19T08:27:19.899 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-11-19T08:27:20.099 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-11-19T08:27:20.978 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-11-19T08:27:21.758 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-11-19T08:27:22.541 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-11-19T08:27:23.449 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-11-19T08:27:24.373 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-11-19T08:27:24.572 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-11-19T08:27:24.770 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-11-19T08:27:24.970 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-11-19T08:27:25.114 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-11-19T08:27:25.157 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-11-19T08:27:25.354 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-11-19T08:27:25.551 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-11-19T08:27:25.739 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-11-19T08:27:25.938 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-11-19T08:27:26.125 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-11-19T08:27:26.313 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-11-19T08:27:26.369 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:26 +0000 (0:00:14.285) 0:02:03.228 ******* 2021-11-19T08:27:26.835 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-11-19T08:27:26.940 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-11-19T08:27:26.942 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "checksum": "b3b89f3dabc35f78585fcbe6afcac330df9308f6", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "69c3df528743245bb36ff3b8fc484a02", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310446.4734147-7829-202388439883565/source", "state": "file", "uid": 0} 2021-11-19T08:27:26.963 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:26 +0000 (0:00:00.593) 0:02:03.822 ******* 2021-11-19T08:27:27.031 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-11-19T08:27:27.033 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => { "msg": "Host smithi134.front.sepia.ceph.com is running CentOS 8 (n/a)" } 2021-11-19T08:27:27.054 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:27 +0000 (0:00:00.091) 0:02:03.913 ******* 2021-11-19T08:27:27.155 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-11-19T08:27:27.157 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-11-19T08:27:27.178 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:27 +0000 (0:00:00.123) 0:02:04.037 ******* 2021-11-19T08:27:27.248 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-11-19T08:27:27.249 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:27.269 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:27 +0000 (0:00:00.091) 0:02:04.128 ******* 2021-11-19T08:27:27.598 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-11-19T08:27:27.600 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.098333", "end": "2021-11-19 08:27:27.590050", "rc": 0, "start": "2021-11-19 08:27:27.491717", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-11-19T08:27:27.620 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:27 +0000 (0:00:00.350) 0:02:04.479 ******* 2021-11-19T08:27:27.710 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-11-19T08:27:27.712 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:27.734 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:27 +0000 (0:00:00.114) 0:02:04.593 ******* 2021-11-19T08:27:27.804 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-11-19T08:27:27.806 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:27.830 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:27 +0000 (0:00:00.095) 0:02:04.689 ******* 2021-11-19T08:27:27.900 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:27.922 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:27 +0000 (0:00:00.091) 0:02:04.781 ******* 2021-11-19T08:27:27.991 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-11-19T08:27:27.993 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:28.014 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:28 +0000 (0:00:00.092) 0:02:04.873 ******* 2021-11-19T08:27:28.094 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-11-19T08:27:28.096 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:28.117 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:28 +0000 (0:00:00.103) 0:02:04.976 ******* 2021-11-19T08:27:28.186 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-11-19T08:27:28.188 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:28.209 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:28 +0000 (0:00:00.091) 0:02:05.067 ******* 2021-11-19T08:27:28.288 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-11-19T08:27:28.290 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi134.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-19T08:27:28.311 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:28 +0000 (0:00:00.102) 0:02:05.170 ******* 2021-11-19T08:27:28.381 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:28.403 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:28 +0000 (0:00:00.092) 0:02:05.262 ******* 2021-11-19T08:27:28.483 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-11-19T08:27:28.485 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:28.505 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:28 +0000 (0:00:00.101) 0:02:05.364 ******* 2021-11-19T08:27:28.522 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:28.523 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-11-19T08:27:28.576 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:28.598 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:28 +0000 (0:00:00.092) 0:02:05.457 ******* 2021-11-19T08:27:28.667 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-11-19T08:27:28.668 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-19T08:27:28.694 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:28 +0000 (0:00:00.095) 0:02:05.552 ******* 2021-11-19T08:27:28.764 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-11-19T08:27:28.766 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:28.787 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:28 +0000 (0:00:00.093) 0:02:05.646 ******* 2021-11-19T08:27:28.868 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi134.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-19T08:27:28.898 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:28 +0000 (0:00:00.111) 0:02:05.757 ******* 2021-11-19T08:27:28.968 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:28.991 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:28 +0000 (0:00:00.092) 0:02:05.849 ******* 2021-11-19T08:27:29.072 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:29.094 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:29 +0000 (0:00:00.102) 0:02:05.952 ******* 2021-11-19T08:27:29.163 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-11-19T08:27:29.165 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:29.185 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:29 +0000 (0:00:00.091) 0:02:06.044 ******* 2021-11-19T08:27:29.254 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-11-19T08:27:29.256 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:29.280 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:29 +0000 (0:00:00.094) 0:02:06.139 ******* 2021-11-19T08:27:29.342 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-11-19T08:27:29.344 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-19T08:27:29.364 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:29 +0000 (0:00:00.084) 0:02:06.223 ******* 2021-11-19T08:27:29.435 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:29.456 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:29 +0000 (0:00:00.091) 0:02:06.314 ******* 2021-11-19T08:27:29.545 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:29.567 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:29 +0000 (0:00:00.111) 0:02:06.426 ******* 2021-11-19T08:27:29.636 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-11-19T08:27:29.638 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:29.658 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:29 +0000 (0:00:00.091) 0:02:06.517 ******* 2021-11-19T08:27:29.738 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-11-19T08:27:29.740 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:29.760 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:29 +0000 (0:00:00.101) 0:02:06.618 ******* 2021-11-19T08:27:29.829 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-11-19T08:27:29.830 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:29.854 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:29 +0000 (0:00:00.094) 0:02:06.713 ******* 2021-11-19T08:27:29.925 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-11-19T08:27:29.927 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:29.948 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:29 +0000 (0:00:00.093) 0:02:06.807 ******* 2021-11-19T08:27:30.018 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-11-19T08:27:30.019 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-19T08:27:30.056 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:30 +0000 (0:00:00.107) 0:02:06.914 ******* 2021-11-19T08:27:30.128 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-11-19T08:27:30.130 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:30.157 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:30 +0000 (0:00:00.102) 0:02:07.016 ******* 2021-11-19T08:27:30.218 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** changed: [smithi023.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-11-19T08:27:30.229 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-11-19T08:27:30.231 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:30.253 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:30 +0000 (0:00:00.095) 0:02:07.111 ******* 2021-11-19T08:27:30.426 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:30.428 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-11-19T08:27:30.521 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-11-19T08:27:30.523 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-19T08:27:30.544 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:30 +0000 (0:00:00.291) 0:02:07.403 ******* 2021-11-19T08:27:30.660 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi023.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-11-19T08:27:30.851 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-11-19T08:27:31.064 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-11-19T08:27:31.275 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-11-19T08:27:31.360 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-11-19T08:27:31.362 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-11-19T08:27:31.468 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi023.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-11-19T08:27:31.677 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-11-19T08:27:31.813 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-11-19T08:27:31.816 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-11-19T08:27:31.837 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:31 +0000 (0:00:01.293) 0:02:08.696 ******* 2021-11-19T08:27:31.877 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-11-19T08:27:31.879 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-11-19T08:27:32.077 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-11-19T08:27:32.277 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-11-19T08:27:32.331 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:32 +0000 (0:00:00.493) 0:02:09.190 ******* 2021-11-19T08:27:32.350 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-11-19T08:27:32.352 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-11-19T08:27:32.792 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-11-19T08:27:32.817 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:32 +0000 (0:00:00.485) 0:02:09.676 ******* 2021-11-19T08:27:32.887 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-11-19T08:27:32.889 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:32.910 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:32 +0000 (0:00:00.093) 0:02:09.769 ******* 2021-11-19T08:27:32.917 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-11-19T08:27:32.919 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "checksum": "b3b89f3dabc35f78585fcbe6afcac330df9308f6", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "69c3df528743245bb36ff3b8fc484a02", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310452.4228988-8522-205947549099408/source", "state": "file", "uid": 0} 2021-11-19T08:27:32.945 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:32 +0000 (0:00:00.034) 0:02:09.804 ******* 2021-11-19T08:27:32.973 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-11-19T08:27:32.975 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:32.999 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:32 +0000 (0:00:00.054) 0:02:09.858 ******* 2021-11-19T08:27:33.018 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-11-19T08:27:33.019 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => { "msg": "Host smithi023.front.sepia.ceph.com is running CentOS 8 (n/a)" } 2021-11-19T08:27:33.043 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.043) 0:02:09.902 ******* 2021-11-19T08:27:33.071 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-11-19T08:27:33.073 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.098 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.054) 0:02:09.957 ******* 2021-11-19T08:27:33.117 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-11-19T08:27:33.118 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-11-19T08:27:33.142 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.043) 0:02:10.001 ******* 2021-11-19T08:27:33.171 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-11-19T08:27:33.172 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.196 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.053) 0:02:10.054 ******* 2021-11-19T08:27:33.215 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-11-19T08:27:33.217 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.241 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.045) 0:02:10.100 ******* 2021-11-19T08:27:33.268 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-11-19T08:27:33.270 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi134.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.280 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.281 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.304 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.062) 0:02:10.162 ******* 2021-11-19T08:27:33.384 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-11-19T08:27:33.385 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.406 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.102) 0:02:10.265 ******* 2021-11-19T08:27:33.485 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-11-19T08:27:33.486 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.507 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.100) 0:02:10.366 ******* 2021-11-19T08:27:33.565 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-11-19T08:27:33.567 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.084279", "end": "2021-11-19 08:27:33.562051", "rc": 0, "start": "2021-11-19 08:27:33.477772", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-11-19T08:27:33.591 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.084) 0:02:10.450 ******* 2021-11-19T08:27:33.598 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-11-19T08:27:33.600 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.624 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.033) 0:02:10.483 ******* 2021-11-19T08:27:33.652 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-11-19T08:27:33.653 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.675 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.051) 0:02:10.534 ******* 2021-11-19T08:27:33.693 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-11-19T08:27:33.695 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.717 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.041) 0:02:10.575 ******* 2021-11-19T08:27:33.744 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-11-19T08:27:33.746 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.769 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.051) 0:02:10.627 ******* 2021-11-19T08:27:33.787 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-11-19T08:27:33.788 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.821 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.052) 0:02:10.680 ******* 2021-11-19T08:27:33.842 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.865 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.043) 0:02:10.724 ******* 2021-11-19T08:27:33.934 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-11-19T08:27:33.936 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:33.957 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:33 +0000 (0:00:00.092) 0:02:10.816 ******* 2021-11-19T08:27:34.028 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-11-19T08:27:34.030 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:34.051 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:34 +0000 (0:00:00.093) 0:02:10.910 ******* 2021-11-19T08:27:34.121 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-11-19T08:27:34.126 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:34.144 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:34 +0000 (0:00:00.092) 0:02:11.002 ******* 2021-11-19T08:27:34.223 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-11-19T08:27:34.225 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-19T08:27:34.245 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:34 +0000 (0:00:00.101) 0:02:11.104 ******* 2021-11-19T08:27:34.324 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:34.346 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:34 +0000 (0:00:00.101) 0:02:11.205 ******* 2021-11-19T08:27:34.425 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-11-19T08:27:34.427 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:34.447 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:34 +0000 (0:00:00.100) 0:02:11.306 ******* 2021-11-19T08:27:34.507 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-11-19T08:27:34.508 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.493281", "end": "2021-11-19 08:27:34.500371", "rc": 0, "start": "2021-11-19 08:27:34.007090", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:27:34.535 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:34 +0000 (0:00:00.087) 0:02:11.394 ******* 2021-11-19T08:27:34.548 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-11-19T08:27:34.550 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:34.572 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:34 +0000 (0:00:00.037) 0:02:11.431 ******* 2021-11-19T08:27:34.652 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-11-19T08:27:34.654 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-19T08:27:34.674 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:34 +0000 (0:00:00.101) 0:02:11.533 ******* 2021-11-19T08:27:34.746 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-11-19T08:27:34.748 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:34.773 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:34 +0000 (0:00:00.098) 0:02:11.631 ******* 2021-11-19T08:27:34.853 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-11-19T08:27:34.855 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-19T08:27:34.876 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:34 +0000 (0:00:00.103) 0:02:11.735 ******* 2021-11-19T08:27:34.945 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-11-19T08:27:34.946 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:34.968 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:34 +0000 (0:00:00.091) 0:02:11.826 ******* 2021-11-19T08:27:35.036 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-11-19T08:27:35.038 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:35.058 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:35 +0000 (0:00:00.090) 0:02:11.917 ******* 2021-11-19T08:27:35.127 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-11-19T08:27:35.129 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:35.150 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:35 +0000 (0:00:00.091) 0:02:12.009 ******* 2021-11-19T08:27:35.219 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-11-19T08:27:35.220 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:35.242 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:35 +0000 (0:00:00.092) 0:02:12.101 ******* 2021-11-19T08:27:35.311 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-11-19T08:27:35.313 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-19T08:27:35.334 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:35 +0000 (0:00:00.091) 0:02:12.192 ******* 2021-11-19T08:27:35.403 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:35.426 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:35 +0000 (0:00:00.092) 0:02:12.285 ******* 2021-11-19T08:27:35.496 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-11-19T08:27:35.498 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:35.521 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:35 +0000 (0:00:00.094) 0:02:12.380 ******* 2021-11-19T08:27:35.591 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-11-19T08:27:35.592 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:35.611 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:35 +0000 (0:00:00.090) 0:02:12.470 ******* 2021-11-19T08:27:35.682 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-11-19T08:27:35.685 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:35.704 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:35 +0000 (0:00:00.092) 0:02:12.563 ******* 2021-11-19T08:27:35.785 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-11-19T08:27:35.786 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:35.806 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:35 +0000 (0:00:00.101) 0:02:12.665 ******* 2021-11-19T08:27:35.895 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-11-19T08:27:35.897 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:35.920 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:35 +0000 (0:00:00.113) 0:02:12.779 ******* 2021-11-19T08:27:35.988 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-11-19T08:27:35.990 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-11-19T08:27:36.010 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:36 +0000 (0:00:00.090) 0:02:12.869 ******* 2021-11-19T08:27:36.079 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-11-19T08:27:36.081 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:36.100 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:36 +0000 (0:00:00.090) 0:02:12.959 ******* 2021-11-19T08:27:36.169 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:36.192 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:36 +0000 (0:00:00.091) 0:02:13.050 ******* 2021-11-19T08:27:36.462 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-11-19T08:27:36.464 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-19T08:27:36.488 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:36 +0000 (0:00:00.296) 0:02:13.347 ******* 2021-11-19T08:27:37.036 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-11-19T08:27:37.038 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-11-19T08:27:37.504 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-11-19T08:27:37.528 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:37 +0000 (0:00:01.040) 0:02:14.387 ******* 2021-11-19T08:27:38.081 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-11-19T08:27:38.082 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-11-19T08:27:38.531 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-11-19T08:27:38.555 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:38 +0000 (0:00:01.026) 0:02:15.414 ******* 2021-11-19T08:27:38.635 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-11-19T08:27:38.636 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:38.659 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:38 +0000 (0:00:00.104) 0:02:15.518 ******* 2021-11-19T08:27:38.722 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-11-19T08:27:38.724 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:38.744 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:38 +0000 (0:00:00.084) 0:02:15.603 ******* 2021-11-19T08:27:38.813 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:38.835 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:38 +0000 (0:00:00.091) 0:02:15.694 ******* 2021-11-19T08:27:38.904 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-11-19T08:27:38.906 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:38.926 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:38 +0000 (0:00:00.091) 0:02:15.785 ******* 2021-11-19T08:27:38.996 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-11-19T08:27:38.997 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:27:39.019 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:39 +0000 (0:00:00.092) 0:02:15.878 ******* 2021-11-19T08:27:39.025 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-11-19T08:27:39.027 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:39.027 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:39.051 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:39 +0000 (0:00:00.032) 0:02:15.910 ******* 2021-11-19T08:27:39.088 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-11-19T08:27:39.090 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:39.113 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:39 +0000 (0:00:00.061) 0:02:15.971 ******* 2021-11-19T08:27:39.120 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-11-19T08:27:39.121 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:39.145 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:39 +0000 (0:00:00.031) 0:02:16.003 ******* 2021-11-19T08:27:39.173 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-11-19T08:27:39.174 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:39.197 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:39 +0000 (0:00:00.052) 0:02:16.056 ******* 2021-11-19T08:27:39.215 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-11-19T08:27:39.217 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:39.240 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:39 +0000 (0:00:00.042) 0:02:16.099 ******* 2021-11-19T08:27:39.269 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-11-19T08:27:39.271 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:39.293 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:39 +0000 (0:00:00.053) 0:02:16.152 ******* 2021-11-19T08:27:39.311 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-11-19T08:27:39.314 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:39.335 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:39 +0000 (0:00:00.041) 0:02:16.194 ******* 2021-11-19T08:27:39.362 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-11-19T08:27:39.364 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:39.388 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:39 +0000 (0:00:00.053) 0:02:16.247 ******* 2021-11-19T08:27:39.406 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-11-19T08:27:39.407 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:39.429 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:39 +0000 (0:00:00.040) 0:02:16.288 ******* 2021-11-19T08:27:39.498 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-11-19T08:27:39.500 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:39.520 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:39 +0000 (0:00:00.090) 0:02:16.379 ******* 2021-11-19T08:27:39.891 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-11-19T08:27:39.919 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:39 +0000 (0:00:00.399) 0:02:16.778 ******* 2021-11-19T08:27:40.506 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-11-19T08:27:40.508 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.787050", "end": "2021-11-19 08:27:40.500319", "rc": 0, "start": "2021-11-19 08:27:39.713269", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:27:40.529 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:40 +0000 (0:00:00.609) 0:02:17.388 ******* 2021-11-19T08:27:41.785 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-11-19T08:27:41.787 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:27:41.808 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:41 +0000 (0:00:01.279) 0:02:18.667 ******* 2021-11-19T08:27:42.332 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-11-19T08:27:42.334 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-11-19T08:27:42.781 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-11-19T08:27:43.212 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-11-19T08:27:43.236 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:43 +0000 (0:00:01.427) 0:02:20.095 ******* 2021-11-19T08:27:43.431 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-11-19T08:27:43.433 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-19T08:27:43.454 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:43 +0000 (0:00:00.217) 0:02:20.313 ******* 2021-11-19T08:27:43.953 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-11-19T08:27:43.954 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-11-19T08:27:44.369 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-11-19T08:27:44.847 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310464.394846-9779-244673189928203/source", "state": "file", "uid": 0} 2021-11-19T08:27:44.904 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:44 +0000 (0:00:01.449) 0:02:21.763 ******* 2021-11-19T08:27:44.931 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-11-19T08:27:44.933 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:27:44.954 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:44 +0000 (0:00:00.050) 0:02:21.813 ******* 2021-11-19T08:27:44.961 INFO:teuthology.task.ansible.out: TASK [common : Include dnf_system vars] **************************************** 2021-11-19T08:27:44.963 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/dnf_systems.yml"], "changed": false} 2021-11-19T08:27:44.984 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:44 +0000 (0:00:00.030) 0:02:21.843 ******* 2021-11-19T08:27:44.991 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-11-19T08:27:44.992 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:45.017 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:45 +0000 (0:00:00.032) 0:02:21.876 ******* 2021-11-19T08:27:45.045 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-11-19T08:27:45.047 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:45.067 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:45 +0000 (0:00:00.049) 0:02:21.926 ******* 2021-11-19T08:27:45.094 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-11-19T08:27:45.096 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:45.123 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:45 +0000 (0:00:00.055) 0:02:21.982 ******* 2021-11-19T08:27:45.153 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-11-19T08:27:45.153 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:45.177 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:45 +0000 (0:00:00.054) 0:02:22.036 ******* 2021-11-19T08:27:45.195 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-11-19T08:27:45.197 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-11-19T08:27:45.219 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:45 +0000 (0:00:00.042) 0:02:22.078 ******* 2021-11-19T08:27:45.267 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-11-19T08:27:45.269 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:45.278 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:45.290 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:45.291 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:45.325 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:45 +0000 (0:00:00.105) 0:02:22.183 ******* 2021-11-19T08:27:45.656 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-11-19T08:27:45.657 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-11-19T08:27:45.681 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:45 +0000 (0:00:00.356) 0:02:22.540 ******* 2021-11-19T08:27:47.145 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-11-19T08:27:47.147 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:27:47.169 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:47 +0000 (0:00:01.487) 0:02:24.028 ******* 2021-11-19T08:27:47.196 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (non-RHEL/CentOS)] ******** 2021-11-19T08:27:47.198 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:47.218 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:47 +0000 (0:00:00.049) 0:02:24.077 ******* 2021-11-19T08:27:47.670 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-11-19T08:27:47.672 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:27:47.694 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:47 +0000 (0:00:00.475) 0:02:24.553 ******* 2021-11-19T08:27:47.702 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-11-19T08:27:47.704 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-11-19T08:27:47.727 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:47 +0000 (0:00:00.032) 0:02:24.586 ******* 2021-11-19T08:27:48.188 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-11-19T08:27:48.190 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-11-19T08:27:48.211 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:48 +0000 (0:00:00.483) 0:02:25.069 ******* 2021-11-19T08:27:48.218 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-11-19T08:27:48.220 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-11-19T08:27:48.263 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi134.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi134.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:48.286 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:48 +0000 (0:00:00.075) 0:02:25.145 ******* 2021-11-19T08:27:48.494 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-11-19T08:27:48.496 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-19T08:27:48.517 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:48 +0000 (0:00:00.230) 0:02:25.376 ******* 2021-11-19T08:27:48.672 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi023.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-11-19T08:27:48.748 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [smithi134.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-11-19T08:27:48.771 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:48 +0000 (0:00:00.254) 0:02:25.630 ******* 2021-11-19T08:27:48.800 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-11-19T08:27:48.802 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:48.835 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:48 +0000 (0:00:00.060) 0:02:25.690 ******* 2021-11-19T08:27:49.136 INFO:teuthology.task.ansible.out: 2021-11-19T08:27:49.138 INFO:teuthology.task.ansible.out:TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi023.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-11-19T08:27:49.167 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:49 +0000 (0:00:00.335) 0:02:26.026 ******* 2021-11-19T08:27:49.315 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-11-19T08:27:49.317 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-11-19T08:27:49.341 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:49 +0000 (0:00:00.173) 0:02:26.199 ******* 2021-11-19T08:27:49.379 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-11-19T08:27:49.381 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-19T08:27:49.405 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:49 +0000 (0:00:00.064) 0:02:26.264 ******* 2021-11-19T08:27:49.888 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi023.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-11-19T08:27:49.985 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-11-19T08:27:49.987 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-11-19 08:24:33 UTC", "ActiveEnterTimestampMonotonic": "19442687", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "var-run.mount systemd-journald.socket systemd-tmpfiles-setup.service sysinit.target tmp.mount basic.target network-online.target time-sync.target -.mount nss-lookup.target local-fs.target system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-11-19 08:24:33 UTC", "AssertTimestampMonotonic": "19419313", "Before": "shutdown.target plymouth-quit.service xdm.service getty@tty1.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-11-19 08:24:33 UTC", "Co 2021-11-19T08:27:49.988 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "19419313", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1256", "ExecMainStartTimestamp": "Fri 2021-11-19 08:24:33 UTC", "ExecMainStartTimestampMonotonic": "19442217", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Fri 2021-11-19 08:24:33 UTC] ; stop_time=[n/a] ; pid=1256 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-11-19 08:24:33 UTC", "InactiveExitTimestampMonotonic": "194426 2021-11-19T08:27:49.990 INFO:teuthology.task.ansible.out:87", "InvocationID": "4f90111534884a808e3213c0d42233a8", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1256", "MemoryAccounting": "yes", "MemoryCurrent": "679936", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectC 2021-11-19T08:27:49.993 INFO:teuthology.task.ansible.out:ontrolGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-11-19 08:24:33 UTC", "StateChangeTimestampMonotonic": "19442687", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-11-19 08:24:33 UTC", "WatchdogTimestampMonotonic": "19442646", "WatchdogUSec": "0"}} 2021-11-19T08:27:50.012 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:50 +0000 (0:00:00.606) 0:02:26.870 ******* 2021-11-19T08:27:50.199 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-11-19T08:27:50.201 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001787", "end": "2021-11-19 08:27:50.192432", "rc": 0, "start": "2021-11-19 08:27:50.190645", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-11-19T08:27:50.222 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:50 +0000 (0:00:00.209) 0:02:27.080 ******* 2021-11-19T08:27:50.335 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-11-19T08:27:50.337 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-11-19T08:27:50.842 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310470.365371-10500-52517154100980/source", "state": "file", "uid": 0} 2021-11-19T08:27:50.867 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:50 +0000 (0:00:00.645) 0:02:27.726 ******* 2021-11-19T08:27:50.906 INFO:teuthology.task.ansible.out: TASK [common : Include dnf_system vars] **************************************** 2021-11-19T08:27:50.908 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/dnf_systems.yml"], "changed": false} 2021-11-19T08:27:50.928 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:50 +0000 (0:00:00.060) 0:02:27.787 ******* 2021-11-19T08:27:51.135 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-11-19T08:27:51.137 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-11-19T08:27:51.157 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:51 +0000 (0:00:00.229) 0:02:28.016 ******* 2021-11-19T08:27:51.196 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi023.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:51.207 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:51.218 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:51.229 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:51.253 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:51 +0000 (0:00:00.095) 0:02:28.111 ******* 2021-11-19T08:27:52.002 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-11-19T08:27:52.004 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:27:52.026 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:52 +0000 (0:00:00.773) 0:02:28.885 ******* 2021-11-19T08:27:53.228 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-11-19T08:27:53.297 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} Friday 19 November 2021 08:27:53 +0000 (0:00:01.225) 0:02:30.111 ******* TASK [common : Install nrpe package and dependencies (non-RHEL/CentOS)] ******** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:53.302 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:53 +0000 (0:00:00.050) 0:02:30.161 ******* 2021-11-19T08:27:53.781 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-11-19T08:27:53.783 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-11-19T08:27:53.802 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:53 +0000 (0:00:00.500) 0:02:30.661 ******* 2021-11-19T08:27:54.292 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-11-19T08:27:54.294 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-11-19T08:27:54.314 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:54 +0000 (0:00:00.511) 0:02:31.173 ******* 2021-11-19T08:27:54.353 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-11-19T08:27:54.355 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:54.365 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-11-19T08:27:54.392 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:54 +0000 (0:00:00.077) 0:02:31.250 ******* 2021-11-19T08:27:54.598 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-11-19T08:27:54.600 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-19T08:27:54.624 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:54 +0000 (0:00:00.232) 0:02:31.483 ******* 2021-11-19T08:27:54.862 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-11-19T08:27:54.864 INFO:teuthology.task.ansible.out:fatal: [smithi023.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-11-19T08:27:54.885 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:54 +0000 (0:00:00.260) 0:02:31.744 ******* 2021-11-19T08:27:54.913 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-11-19T08:27:54.915 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:27:54.935 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:54 +0000 (0:00:00.049) 0:02:31.794 ******* 2021-11-19T08:27:55.421 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-11-19T08:27:55.423 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-11-19T08:27:55.443 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:55 +0000 (0:00:00.507) 0:02:32.302 ******* 2021-11-19T08:27:55.754 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-11-19T08:27:55.755 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-11-19 08:25:03 UTC", "ActiveEnterTimestampMonotonic": "19343944", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network-online.target local-fs.target var-run.mount systemd-tmpfiles-setup.service -.mount sysinit.target nss-lookup.target basic.target systemd-journald.socket system.slice time-sync.target tmp.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-11-19 08:25:03 UTC", "AssertTimestampMonotonic": "19335506", "Before": "plymouth-quit.service xdm.service shutdown.target getty@tty1.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-11-19 08:25:03 UTC", "Co 2021-11-19T08:27:55.757 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "19335505", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1283", "ExecMainStartTimestamp": "Fri 2021-11-19 08:25:03 UTC", "ExecMainStartTimestampMonotonic": "19343896", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Fri 2021-11-19 08:25:03 UTC] ; stop_time=[n/a] ; pid=1283 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-11-19 08:25:03 UTC", "InactiveExitTimestampMonotonic": "193439 2021-11-19T08:27:55.758 INFO:teuthology.task.ansible.out:44", "InvocationID": "91e751164d524542979eecaa55c8f4b9", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1283", "MemoryAccounting": "yes", "MemoryCurrent": "671744", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectC 2021-11-19T08:27:55.760 INFO:teuthology.task.ansible.out:ontrolGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-11-19 08:25:03 UTC", "StateChangeTimestampMonotonic": "19343944", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-11-19 08:25:03 UTC", "WatchdogTimestampMonotonic": "19343943", "WatchdogUSec": "0"}} 2021-11-19T08:27:55.778 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:55 +0000 (0:00:00.335) 0:02:32.637 ******* 2021-11-19T08:27:55.983 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-11-19T08:27:55.985 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001859", "end": "2021-11-19 08:27:55.969690", "rc": 0, "start": "2021-11-19 08:27:55.967831", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-11-19T08:27:56.006 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:56 +0000 (0:00:00.227) 0:02:32.864 ******* 2021-11-19T08:27:57.896 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-11-19T08:27:57.898 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:27:57.920 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:27:57 +0000 (0:00:01.914) 0:02:34.779 ******* 2021-11-19T08:28:31.065 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-11-19T08:28:31.067 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-83.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-83.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-11-19T08:28:31.089 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:28:31 +0000 (0:00:33.168) 0:03:07.948 ******* 2021-11-19T08:28:31.808 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-11-19T08:28:31.809 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-11-19T08:28:31.833 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:28:31 +0000 (0:00:00.743) 0:03:08.691 ******* 2021-11-19T08:28:38.204 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-11-19T08:28:38.206 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-3.14.3-83.el8.noarch", "Installed: selinux-policy-targeted-3.14.3-83.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-11-19T08:28:38.228 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:28:38 +0000 (0:00:06.395) 0:03:15.087 ******* 2021-11-19T08:28:38.699 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-11-19T08:28:38.701 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-11-19T08:28:38.722 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:28:38 +0000 (0:00:00.494) 0:03:15.581 ******* 2021-11-19T08:28:47.788 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-11-19T08:28:47.790 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.759922", "end": "2021-11-19 08:28:47.774640", "failed_when_result": false, "rc": 0, "start": "2021-11-19 08:28:32.014718", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-11-19T08:28:47.811 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:28:47 +0000 (0:00:09.089) 0:03:24.670 ******* 2021-11-19T08:28:48.268 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-11-19T08:28:48.270 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-11-19T08:28:48.291 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:28:48 +0000 (0:00:00.479) 0:03:25.150 ******* 2021-11-19T08:28:48.496 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-11-19T08:28:48.498 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.017704", "end": "2021-11-19 08:28:48.492816", "rc": 0, "start": "2021-11-19 08:28:48.475112", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:28:48.519 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:28:48 +0000 (0:00:00.227) 0:03:25.378 ******* 2021-11-19T08:28:48.712 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-11-19T08:28:48.715 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.008886", "end": "2021-11-19 08:28:48.708404", "rc": 0, "start": "2021-11-19 08:28:48.699518", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:28:48.734 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:28:48 +0000 (0:00:00.215) 0:03:25.593 ******* 2021-11-19T08:28:55.335 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-11-19T08:28:55.337 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.411111", "end": "2021-11-19 08:28:55.323439", "failed_when_result": false, "rc": 0, "start": "2021-11-19 08:28:38.912328", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-11-19T08:28:55.358 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:28:55 +0000 (0:00:06.623) 0:03:32.217 ******* 2021-11-19T08:28:55.831 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-11-19T08:28:55.833 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-11-19T08:28:55.854 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:28:55 +0000 (0:00:00.495) 0:03:32.713 ******* 2021-11-19T08:28:56.060 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-11-19T08:28:56.062 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.008318", "end": "2021-11-19 08:28:56.053668", "rc": 0, "start": "2021-11-19 08:28:56.045350", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:28:56.083 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:28:56 +0000 (0:00:00.229) 0:03:32.942 ******* 2021-11-19T08:28:56.286 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-11-19T08:28:56.287 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.007827", "end": "2021-11-19 08:28:56.277700", "rc": 0, "start": "2021-11-19 08:28:56.269873", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:28:56.308 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:28:56 +0000 (0:00:00.224) 0:03:33.167 ******* 2021-11-19T08:29:04.607 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-11-19T08:29:04.608 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.678242", "end": "2021-11-19 08:29:04.593515", "rc": 0, "start": "2021-11-19 08:28:48.915273", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:29:04.632 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:04 +0000 (0:00:08.323) 0:03:41.490 ******* 2021-11-19T08:29:04.848 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-11-19T08:29:04.849 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-11-19T08:29:04.872 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:04 +0000 (0:00:00.240) 0:03:41.730 ******* 2021-11-19T08:29:04.951 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} Friday 19 November 2021 08:29:04 +0000 (0:00:00.075) 0:03:41.806 ******* 2021-11-19T08:29:05.000 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-11-19T08:29:05.002 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-11-19T08:29:05.034 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:05 +0000 (0:00:00.086) 0:03:41.892 ******* 2021-11-19T08:29:05.085 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-11-19T08:29:05.087 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-19T08:29:05.089 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-19T08:29:05.110 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:05 +0000 (0:00:00.076) 0:03:41.969 ******* 2021-11-19T08:29:05.171 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-11-19T08:29:05.173 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-19T08:29:05.174 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-19T08:29:05.205 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:05 +0000 (0:00:00.094) 0:03:42.064 ******* 2021-11-19T08:29:05.405 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-11-19T08:29:05.406 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-19T08:29:05.432 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:05 +0000 (0:00:00.227) 0:03:42.291 ******* 2021-11-19T08:29:05.632 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-11-19T08:29:05.654 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:05 +0000 (0:00:00.221) 0:03:42.513 ******* 2021-11-19T08:29:05.681 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-11-19T08:29:05.683 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:29:05.705 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:05 +0000 (0:00:00.051) 0:03:42.564 ******* 2021-11-19T08:29:05.734 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-11-19T08:29:05.736 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:29:05.766 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:05 +0000 (0:00:00.060) 0:03:42.625 ******* 2021-11-19T08:29:05.989 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-11-19T08:29:05.991 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-11-19T08:29:06.024 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:06 +0000 (0:00:00.258) 0:03:42.883 ******* 2021-11-19T08:29:06.572 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-11-19T08:29:06.573 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-11-19T08:29:06.593 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:06 +0000 (0:00:00.569) 0:03:43.452 ******* 2021-11-19T08:29:06.631 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi134"}, "changed": false} 2021-11-19T08:29:06.653 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:06 +0000 (0:00:00.059) 0:03:43.512 ******* 2021-11-19T08:29:07.498 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-11-19T08:29:07.500 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi134.front.sepia.ceph.com", "ansible_hostname": "smithi134", "ansible_nodename": "smithi134"}, "changed": true, "name": "smithi134"} 2021-11-19T08:29:07.520 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:07 +0000 (0:00:00.867) 0:03:44.379 ******* 2021-11-19T08:29:07.975 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-11-19T08:29:07.977 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-11-19T08:29:07.998 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:07 +0000 (0:00:00.476) 0:03:44.856 ******* 2021-11-19T08:29:08.466 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-11-19T08:29:08.468 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-11-19T08:29:08.490 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:08 +0000 (0:00:00.492) 0:03:45.348 ******* 2021-11-19T08:29:09.023 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-11-19T08:29:09.024 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-11-19T08:29:09.045 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:09 +0000 (0:00:00.555) 0:03:45.904 ******* 2021-11-19T08:29:09.533 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-11-19T08:29:09.534 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-11-19T08:29:09.555 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:09 +0000 (0:00:00.509) 0:03:46.414 ******* 2021-11-19T08:29:09.750 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-19T08:29:09.752 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-19T08:29:09.773 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:09 +0000 (0:00:00.217) 0:03:46.632 ******* 2021-11-19T08:29:09.977 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-19T08:29:09.978 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-11-19T08:29:09.998 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:09 +0000 (0:00:00.225) 0:03:46.857 ******* 2021-11-19T08:29:10.222 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-19T08:29:10.223 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-11-19T08:29:10.244 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:10 +0000 (0:00:00.246) 0:03:47.103 ******* 2021-11-19T08:29:10.721 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-11-19T08:29:10.722 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-11-19T08:29:10.748 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:10 +0000 (0:00:00.503) 0:03:47.607 ******* 2021-11-19T08:29:11.220 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-19T08:29:11.222 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-11-19T08:29:11.242 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:11 +0000 (0:00:00.494) 0:03:48.101 ******* 2021-11-19T08:29:12.685 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-11-19T08:29:12.687 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.180568", "end": "2021-11-19 08:29:12.677870", "rc": 0, "start": "2021-11-19 08:28:56.497302", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:29:12.708 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:12 +0000 (0:00:01.465) 0:03:49.567 ******* 2021-11-19T08:29:12.913 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-11-19T08:29:12.915 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-11-19T08:29:12.936 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:12 +0000 (0:00:00.227) 0:03:49.794 ******* 2021-11-19T08:29:12.963 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-11-19T08:29:12.965 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-11-19T08:29:12.986 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:12 +0000 (0:00:00.050) 0:03:49.845 ******* 2021-11-19T08:29:13.036 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-11-19T08:29:13.037 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-11-19T08:29:13.059 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:13 +0000 (0:00:00.072) 0:03:49.917 ******* 2021-11-19T08:29:13.110 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "Ce 2021-11-19T08:29:13.111 INFO:teuthology.task.ansible.out:ntOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-19T08:29:13.137 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:13 +0000 (0:00:00.079) 0:03:49.996 ******* 2021-11-19T08:29:13.176 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-11-19T08:29:13.177 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"apache-arrow": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/R 2021-11-19T08:29:13.179 INFO:teuthology.task.ansible.out:PM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-11-19T08:29:13.199 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:13 +0000 (0:00:00.062) 0:03:50.058 ******* 2021-11-19T08:29:13.393 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-11-19T08:29:13.396 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-11-19T08:29:13.415 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:13 +0000 (0:00:00.215) 0:03:50.274 ******* 2021-11-19T08:29:13.619 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-11-19T08:29:13.621 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-11-19T08:29:13.640 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:13 +0000 (0:00:00.225) 0:03:50.499 ******* 2021-11-19T08:29:13.667 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-11-19T08:29:13.669 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:29:13.689 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:13 +0000 (0:00:00.048) 0:03:50.548 ******* 2021-11-19T08:29:13.715 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-11-19T08:29:13.717 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:29:13.737 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:13 +0000 (0:00:00.047) 0:03:50.595 ******* 2021-11-19T08:29:13.919 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-19T08:29:13.921 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.478491", "end": "2021-11-19 08:29:13.903106", "rc": 0, "start": "2021-11-19 08:29:11.424615", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:29:13.943 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:13 +0000 (0:00:00.206) 0:03:50.802 ******* 2021-11-19T08:29:13.970 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-11-19T08:29:13.972 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-11-19T08:29:13.997 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:13 +0000 (0:00:00.054) 0:03:50.856 ******* 2021-11-19T08:29:14.150 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-11-19T08:29:14.151 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rvf /etc/yum.repos.d/*-Stream-*", "delta": "0:00:00.002732", "end": "2021-11-19 08:29:14.144120", "rc": 0, "start": "2021-11-19 08:29:14.141388", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:29:14.174 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:14 +0000 (0:00:00.177) 0:03:51.033 ******* 2021-11-19T08:29:14.213 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:29:14.215 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"dash_stream": "-stream"}, "changed": false} 2021-11-19T08:29:14.235 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:14 +0000 (0:00:00.060) 0:03:51.094 ******* 2021-11-19T08:29:14.283 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-11-19T08:29:14.284 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml) => {"ansible_facts": {"packages_to_upgrade": ["systemd"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"} 2021-11-19T08:29:14.306 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:14 +0000 (0:00:00.070) 0:03:51.164 ******* 2021-11-19T08:29:14.540 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-11-19T08:29:14.542 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-11-19T08:29:14.562 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:14 +0000 (0:00:00.256) 0:03:51.421 ******* 2021-11-19T08:29:14.601 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi023"}, "changed": false} 2021-11-19T08:29:14.622 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:14 +0000 (0:00:00.059) 0:03:51.481 ******* 2021-11-19T08:29:14.774 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-11-19T08:29:14.776 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "be4ef7d7f1f4743a5048cab6a4dced5e5d1e5b14", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2021-11-19T08:29:15.067 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-11-19T08:29:15.068 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi023.front.sepia.ceph.com", "ansible_hostname": "smithi023", "ansible_nodename": "smithi023"}, "changed": true, "name": "smithi023"} 2021-11-19T08:29:15.089 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:15 +0000 (0:00:00.467) 0:03:51.948 ******* 2021-11-19T08:29:15.222 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-11-19T08:29:15.224 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "5a66a2928a1d64d8d11230f172eb6b696c57c58c", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2021-11-19T08:29:15.559 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-11-19T08:29:15.561 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-11-19T08:29:15.581 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:15 +0000 (0:00:00.491) 0:03:52.440 ******* 2021-11-19T08:29:15.662 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-11-19T08:29:15.664 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "12b6934a87ad915f812630b0a7e54663770cd059", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2021-11-19T08:29:16.073 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-11-19T08:29:16.075 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-11-19T08:29:16.096 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:16 +0000 (0:00:00.514) 0:03:52.955 ******* 2021-11-19T08:29:16.160 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-11-19T08:29:16.162 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "c36ac65b18cdf8071cdb7bd4e46c5639be96cbac", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "md5sum": "76ff786610d5a5d6d02f2e8fd5311b89", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310555.6838417-21074-99770782667681/source", "state": "file", "uid": 0} 2021-11-19T08:29:16.187 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:16 +0000 (0:00:00.091) 0:03:53.046 ******* 2021-11-19T08:29:16.660 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-11-19T08:29:16.662 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-11-19T08:29:16.683 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:16 +0000 (0:00:00.495) 0:03:53.542 ******* 2021-11-19T08:29:16.691 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-11-19T08:29:16.692 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-11-19T08:29:17.154 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'apache-arrow', 'value': {'name': 'Apache Arrow', 'baseurl': 'https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch', 'enabled': 1, 'gpgcheck': 1}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "86b924db6e383ddad921a207c09a708e6cca6a67", "dest": "/etc/yum.repos.d/apache-arrow.repo", "gid": 0, "group": "root", "item": {"key": "apache-arrow", "value": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}}, "md5sum": "183d0889f7213a1e38c7f48965d7549e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 215, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310556.6997523-21362-197119981524269/source", "state": "file", "uid": 0} 2021-11-19T08:29:17.179 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:17 +0000 (0:00:00.496) 0:03:54.038 ******* 2021-11-19T08:29:17.186 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-11-19T08:29:17.187 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-11-19T08:29:17.211 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:17 +0000 (0:00:00.031) 0:03:54.070 ******* 2021-11-19T08:29:17.418 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-19T08:29:17.419 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-19T08:29:17.440 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:17 +0000 (0:00:00.228) 0:03:54.299 ******* 2021-11-19T08:29:17.648 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-19T08:29:17.650 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-11-19T08:29:17.679 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:17 +0000 (0:00:00.238) 0:03:54.537 ******* 2021-11-19T08:29:17.686 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-11-19T08:29:17.688 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-11-19T08:29:17.919 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-19T08:29:17.920 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-11-19T08:29:17.946 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:17 +0000 (0:00:00.267) 0:03:54.805 ******* 2021-11-19T08:29:18.099 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-11-19T08:29:18.101 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-11-19T08:29:18.445 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-11-19T08:29:18.467 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:18 +0000 (0:00:00.521) 0:03:55.326 ******* 2021-11-19T08:29:18.537 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-11-19T08:29:18.538 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-11-19T08:29:18.962 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-19T08:29:18.964 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-11-19T08:29:18.984 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:18 +0000 (0:00:00.516) 0:03:55.843 ******* 2021-11-19T08:29:19.033 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "md5sum": "d96965513d5dae48e1f179ee52900b67", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310558.571744-21501-17516447577266/source", "state": "file", "uid": 0} 2021-11-19T08:29:19.066 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:19 +0000 (0:00:00.082) 0:03:55.925 ******* 2021-11-19T08:29:19.507 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-11-19T08:29:19.509 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.235263", "end": "2021-11-19 08:29:19.496531", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-11-19 08:29:19.261268", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'comma 2021-11-19T08:29:19.511 INFO:teuthology.task.ansible.out:nd_warnings=False' in ansible.cfg to get rid of this message. 2021-11-19T08:29:19.530 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:19 +0000 (0:00:00.463) 0:03:56.389 ******* 2021-11-19T08:29:21.782 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-19T08:29:21.784 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.595713", "end": "2021-11-19 08:29:21.772641", "rc": 0, "start": "2021-11-19 08:29:19.176928", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:29:21.806 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:21 +0000 (0:00:02.275) 0:03:58.665 ******* 2021-11-19T08:29:22.022 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-11-19T08:29:22.024 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rvf /etc/yum.repos.d/*-Stream-*", "delta": "0:00:00.002933", "end": "2021-11-19 08:29:22.017748", "rc": 0, "start": "2021-11-19 08:29:22.014815", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:29:22.046 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:22 +0000 (0:00:00.240) 0:03:58.905 ******* 2021-11-19T08:29:22.095 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:29:22.097 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"dash_stream": "-stream"}, "changed": false} 2021-11-19T08:29:22.117 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:22 +0000 (0:00:00.070) 0:03:58.976 ******* 2021-11-19T08:29:22.176 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-11-19T08:29:22.178 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml) => {"ansible_facts": {"packages_to_upgrade": ["systemd"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8_stream.yml"} 2021-11-19T08:29:22.206 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:22 +0000 (0:00:00.088) 0:03:59.065 ******* 2021-11-19T08:29:22.695 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-11-19T08:29:22.697 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "be4ef7d7f1f4743a5048cab6a4dced5e5d1e5b14", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2021-11-19T08:29:23.152 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "5a66a2928a1d64d8d11230f172eb6b696c57c58c", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2021-11-19T08:29:23.614 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "12b6934a87ad915f812630b0a7e54663770cd059", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2021-11-19T08:29:24.118 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "c36ac65b18cdf8071cdb7bd4e46c5639be96cbac", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "md5sum": "76ff786610d5a5d6d02f2e8fd5311b89", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310563.6418622-22272-31105238666754/source", "state": "file", "uid": 0} 2021-11-19T08:29:24.145 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:24 +0000 (0:00:01.938) 0:04:01.003 ******* 2021-11-19T08:29:24.657 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* ok: [smithi023.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-11-19T08:29:25.169 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'apache-arrow', 'value': {'name': 'Apache Arrow', 'baseurl': 'https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch', 'enabled': 1, 'gpgcheck': 1}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "86b924db6e383ddad921a207c09a708e6cca6a67", "dest": "/etc/yum.repos.d/apache-arrow.repo", "gid": 0, "group": "root", "item": {"key": "apache-arrow", "value": {"baseurl": "https://apache.jfrog.io/artifactory/arrow/centos/8/$basearch", "enabled": 1, "gpgcheck": 1, "name": "Apache Arrow"}}, "md5sum": "183d0889f7213a1e38c7f48965d7549e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 215, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310564.6874447-22555-190323430977715/source", "state": "file", "uid": 0} 2021-11-19T08:29:25.193 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:25 +0000 (0:00:01.048) 0:04:02.052 ******* 2021-11-19T08:29:25.705 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-11-19T08:29:25.707 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-11-19T08:29:26.182 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-11-19T08:29:26.645 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-11-19T08:29:26.688 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-11-19T08:29:26.690 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:29:26.712 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:26 +0000 (0:00:01.519) 0:04:03.571 ******* 2021-11-19T08:29:27.163 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-11-19T08:29:27.165 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": true, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "md5sum": "d96965513d5dae48e1f179ee52900b67", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310566.6732094-22697-189980585422064/source", "state": "file", "uid": 0} 2021-11-19T08:29:27.201 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:27 +0000 (0:00:00.488) 0:04:04.059 ******* 2021-11-19T08:29:27.669 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-11-19T08:29:27.669 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/release.asc"} 2021-11-19T08:29:27.712 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-11-19T08:29:27.714 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.300850", "end": "2021-11-19 08:29:27.701239", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-11-19 08:29:27.400389", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-11-19T08:29:27.735 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:27 +0000 (0:00:00.534) 0:04:04.594 ******* 2021-11-19T08:29:28.238 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-11-19T08:29:28.240 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/autobuild.asc"} 2021-11-19T08:29:29.579 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=https://dist.apache.org/repos/dist/dev/arrow/KEYS) => {"ansible_loop_var": "item", "changed": true, "item": "https://dist.apache.org/repos/dist/dev/arrow/KEYS"} 2021-11-19T08:29:29.604 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:29 +0000 (0:00:01.868) 0:04:06.462 ******* 2021-11-19T08:29:30.001 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-11-19T08:29:30.002 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum module rather than running 'yum'. If you need to use command because yum is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.184854", "end": "2021-11-19 08:29:29.988492", "rc": 0, "start": "2021-11-19 08:29:29.803638", "stderr": "", "stderr_lines": [], "stdout": "57 files removed", "stdout_lines": ["57 files removed"]} 2021-11-19T08:29:30.034 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:30 +0000 (0:00:00.430) 0:04:06.893 ******* 2021-11-19T08:29:30.075 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:29:30.077 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-11-19T08:29:30.104 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:30 +0000 (0:00:00.069) 0:04:06.962 ******* 2021-11-19T08:29:30.146 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-11-19T08:29:30.147 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:29:30.170 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:30 +0000 (0:00:00.065) 0:04:07.029 ******* 2021-11-19T08:29:30.209 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-11-19T08:29:30.211 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:29:30.231 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:30 +0000 (0:00:00.061) 0:04:07.090 ******* 2021-11-19T08:29:30.436 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-11-19T08:29:30.438 INFO:teuthology.task.ansible.out:fatal: [smithi134.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007535", "end": "2021-11-19 08:29:30.425016", "msg": "non-zero return code", "rc": 1, "start": "2021-11-19 08:29:30.417481", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-11-19T08:29:30.462 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:30 +0000 (0:00:00.230) 0:04:07.321 ******* 2021-11-19T08:29:30.490 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-11-19T08:29:30.492 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:29:30.520 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:30 +0000 (0:00:00.057) 0:04:07.378 ******* 2021-11-19T08:29:35.298 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-11-19T08:29:35.300 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:29:35.321 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:35 +0000 (0:00:04.801) 0:04:12.180 ******* 2021-11-19T08:29:35.878 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-11-19T08:29:35.880 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/release.asc"} 2021-11-19T08:29:36.349 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": true, "item": "https://download.ceph.com/keys/autobuild.asc"} 2021-11-19T08:29:37.772 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=https://dist.apache.org/repos/dist/dev/arrow/KEYS) => {"ansible_loop_var": "item", "changed": true, "item": "https://dist.apache.org/repos/dist/dev/arrow/KEYS"} 2021-11-19T08:29:37.795 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:37 +0000 (0:00:02.474) 0:04:14.654 ******* 2021-11-19T08:29:38.209 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-11-19T08:29:38.211 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.193009", "end": "2021-11-19 08:29:38.199006", "rc": 0, "start": "2021-11-19 08:29:38.005997", "stderr": "", "stderr_lines": [], "stdout": "57 files removed", "stdout_lines": ["57 files removed"]} 2021-11-19T08:29:38.232 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:38 +0000 (0:00:00.436) 0:04:15.090 ******* 2021-11-19T08:29:38.260 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:29:38.261 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-11-19T08:29:38.281 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:38 +0000 (0:00:00.049) 0:04:15.140 ******* 2021-11-19T08:29:38.319 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-11-19T08:29:38.321 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:29:38.341 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:38 +0000 (0:00:00.059) 0:04:15.200 ******* 2021-11-19T08:29:38.379 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-11-19T08:29:38.381 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:29:38.401 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:38 +0000 (0:00:00.060) 0:04:15.260 ******* 2021-11-19T08:29:38.615 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-11-19T08:29:38.617 INFO:teuthology.task.ansible.out:fatal: [smithi023.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007835", "end": "2021-11-19 08:29:38.603654", "msg": "non-zero return code", "rc": 1, "start": "2021-11-19 08:29:38.595819", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-11-19T08:29:38.638 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:38 +0000 (0:00:00.236) 0:04:15.496 ******* 2021-11-19T08:29:38.675 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-11-19T08:29:38.676 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:29:38.697 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:38 +0000 (0:00:00.059) 0:04:15.556 ******* 2021-11-19T08:29:48.577 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-11-19T08:29:48.579 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-11-19T08:29:48.602 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:48 +0000 (0:00:09.904) 0:04:25.461 ******* 2021-11-19T08:29:52.516 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-11-19T08:29:52.538 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:52 +0000 (0:00:03.935) 0:04:29.396 ******* 2021-11-19T08:29:57.782 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-11-19T08:29:57.785 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-11-19T08:29:57.807 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:29:57 +0000 (0:00:05.269) 0:04:34.666 ******* 2021-11-19T08:30:01.152 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-11-19T08:30:01.154 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-11-19T08:30:01.178 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:01 +0000 (0:00:03.370) 0:04:38.037 ******* 2021-11-19T08:30:02.262 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-19T08:30:02.264 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: qemu-kvm-common-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: python3-dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: qemu-kvm-core-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: yum-utils-4.0.21-3.el8.noarch", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: yum-utils-4.0.18-4.el8.noarch"]} 2021-11-19T08:30:02.286 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:02 +0000 (0:00:01.108) 0:04:39.145 ******* 2021-11-19T08:30:02.314 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-11-19T08:30:02.316 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:02.336 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:02 +0000 (0:00:00.049) 0:04:39.195 ******* 2021-11-19T08:30:04.484 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:30:04.506 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:04 +0000 (0:00:02.169) 0:04:41.365 ******* 2021-11-19T08:30:10.654 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-19T08:30:10.656 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: qemu-kvm-common-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: python3-dnf-plugins-core-4.0.21-3.el8.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: qemu-img-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: qemu-kvm-core-15:6.1.0-4.module_el8.6.0+983+a7505f3f.x86_64", "Installed: yum-utils-4.0.21-3.el8.noarch", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.5.0+746+bbd5d70c.x86_64", "Removed: python3-dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: dnf-plugins-core-4.0.18-4.el8.noarch", "Removed: yum-utils-4.0.18-4.el8.noarch"]} 2021-11-19T08:30:10.678 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:10 +0000 (0:00:06.172) 0:04:47.537 ******* 2021-11-19T08:30:10.717 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-11-19T08:30:10.719 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:10.739 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:10 +0000 (0:00:00.060) 0:04:47.598 ******* 2021-11-19T08:30:12.337 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-19T08:30:12.339 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: systemd-pam-239-51.el8.x86_64", "Installed: systemd-239-51.el8.x86_64", "Installed: systemd-libs-239-51.el8.x86_64", "Installed: systemd-udev-239-51.el8.x86_64", "Removed: systemd-239-45.el8.x86_64", "Removed: systemd-libs-239-45.el8.x86_64", "Removed: systemd-pam-239-45.el8.x86_64", "Removed: systemd-udev-239-45.el8.x86_64"]} 2021-11-19T08:30:12.364 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:12 +0000 (0:00:01.625) 0:04:49.223 ******* 2021-11-19T08:30:12.678 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-11-19T08:30:12.680 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target polkit.service system.slice dbus.socket dbus.service sysinit.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target iptables.service ip6tables.service ebtables.service ipset.service nftables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate" 2021-11-19T08:30:12.681 INFO:teuthology.task.ansible.out:: "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": 2021-11-19T08:30:12.683 INFO:teuthology.task.ansible.out:"0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket system.slice sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "nul 2021-11-19T08:30:12.685 INFO:teuthology.task.ansible.out:l", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-11-19T08:30:12.704 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:12 +0000 (0:00:00.339) 0:04:49.563 ******* 2021-11-19T08:30:12.996 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-11-19T08:30:12.998 INFO:teuthology.task.ansible.out:fatal: [smithi134.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-11-19T08:30:13.019 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:13 +0000 (0:00:00.315) 0:04:49.878 ******* 2021-11-19T08:30:13.026 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-11-19T08:30:13.028 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:30:13.051 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:13 +0000 (0:00:00.031) 0:04:49.909 ******* 2021-11-19T08:30:13.669 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-11-19T08:30:13.671 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-11-19T08:30:13.693 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:13 +0000 (0:00:00.642) 0:04:50.552 ******* 2021-11-19T08:30:13.721 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-11-19T08:30:13.723 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:13.743 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:13 +0000 (0:00:00.050) 0:04:50.602 ******* 2021-11-19T08:30:13.771 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-11-19T08:30:13.773 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:13.793 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:13 +0000 (0:00:00.050) 0:04:50.652 ******* 2021-11-19T08:30:13.821 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-11-19T08:30:13.823 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:13.845 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:13 +0000 (0:00:00.051) 0:04:50.704 ******* 2021-11-19T08:30:13.872 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-11-19T08:30:13.874 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:13.895 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:13 +0000 (0:00:00.049) 0:04:50.753 ******* 2021-11-19T08:30:13.923 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-11-19T08:30:13.924 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:13.954 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:13 +0000 (0:00:00.058) 0:04:50.812 ******* 2021-11-19T08:30:13.984 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-11-19T08:30:13.986 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.008 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.054) 0:04:50.867 ******* 2021-11-19T08:30:14.036 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-11-19T08:30:14.059 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.050) 0:04:50.917 ******* 2021-11-19T08:30:14.086 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-11-19T08:30:14.088 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.116 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.057) 0:04:50.975 ******* 2021-11-19T08:30:14.144 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-11-19T08:30:14.144 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.176 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.059) 0:04:51.035 ******* 2021-11-19T08:30:14.217 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-11-19T08:30:14.219 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi134.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.242 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.065) 0:04:51.101 ******* 2021-11-19T08:30:14.270 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-11-19T08:30:14.271 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.294 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.051) 0:04:51.153 ******* 2021-11-19T08:30:14.322 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-11-19T08:30:14.348 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.054) 0:04:51.207 ******* 2021-11-19T08:30:14.378 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-11-19T08:30:14.380 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.400 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.052) 0:04:51.259 ******* 2021-11-19T08:30:14.428 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-11-19T08:30:14.429 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.450 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.049) 0:04:51.309 ******* 2021-11-19T08:30:14.478 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.500 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.050) 0:04:51.359 ******* 2021-11-19T08:30:14.528 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-19T08:30:14.529 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.561 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.060) 0:04:51.419 ******* 2021-11-19T08:30:14.591 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-11-19T08:30:14.592 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.613 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.052) 0:04:51.472 ******* 2021-11-19T08:30:14.640 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-19T08:30:14.642 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.663 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.049) 0:04:51.522 ******* 2021-11-19T08:30:14.691 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.714 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.051) 0:04:51.573 ******* 2021-11-19T08:30:14.744 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.768 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.053) 0:04:51.626 ******* 2021-11-19T08:30:14.796 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-11-19T08:30:14.797 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.821 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.053) 0:04:51.680 ******* 2021-11-19T08:30:14.848 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.871 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.049) 0:04:51.729 ******* 2021-11-19T08:30:14.897 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-11-19T08:30:14.899 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.920 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.049) 0:04:51.778 ******* 2021-11-19T08:30:14.947 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-11-19T08:30:14.949 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:14.969 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:14 +0000 (0:00:00.049) 0:04:51.828 ******* 2021-11-19T08:30:14.996 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-11-19T08:30:14.997 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.020 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.050) 0:04:51.878 ******* 2021-11-19T08:30:15.047 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-11-19T08:30:15.049 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.068 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.048) 0:04:51.927 ******* 2021-11-19T08:30:15.096 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-11-19T08:30:15.098 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.119 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.050) 0:04:51.978 ******* 2021-11-19T08:30:15.148 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-19T08:30:15.149 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.171 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.051) 0:04:52.029 ******* 2021-11-19T08:30:15.200 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-19T08:30:15.202 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.235 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.064) 0:04:52.094 ******* 2021-11-19T08:30:15.264 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-19T08:30:15.265 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.286 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.050) 0:04:52.145 ******* 2021-11-19T08:30:15.314 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-11-19T08:30:15.316 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.336 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.049) 0:04:52.195 ******* 2021-11-19T08:30:15.363 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-19T08:30:15.365 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.385 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.049) 0:04:52.244 ******* 2021-11-19T08:30:15.414 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-19T08:30:15.416 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.436 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.051) 0:04:52.295 ******* 2021-11-19T08:30:15.474 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.498 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.060) 0:04:52.356 ******* 2021-11-19T08:30:15.527 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-11-19T08:30:15.531 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.560 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.062) 0:04:52.418 ******* 2021-11-19T08:30:15.592 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:30:15.593 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.615 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.055) 0:04:52.474 ******* 2021-11-19T08:30:15.643 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.666 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.050) 0:04:52.524 ******* 2021-11-19T08:30:15.693 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-19T08:30:15.695 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.715 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.049) 0:04:52.574 ******* 2021-11-19T08:30:15.742 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-19T08:30:15.744 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.766 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.051) 0:04:52.625 ******* 2021-11-19T08:30:15.811 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-19T08:30:15.813 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:15.834 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:15 +0000 (0:00:00.067) 0:04:52.693 ******* 2021-11-19T08:30:16.052 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-19T08:30:16.054 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-11-19T08:30:16.075 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:16 +0000 (0:00:00.241) 0:04:52.934 ******* 2021-11-19T08:30:16.309 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-19T08:30:16.310 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-19T08:30:16.330 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:16 +0000 (0:00:00.254) 0:04:53.189 ******* 2021-11-19T08:30:16.539 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-19T08:30:16.541 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-11-19T08:30:16.563 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:16 +0000 (0:00:00.233) 0:04:53.422 ******* 2021-11-19T08:30:16.593 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-19T08:30:16.595 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:16.616 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:16 +0000 (0:00:00.052) 0:04:53.475 ******* 2021-11-19T08:30:16.656 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:16.679 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:16 +0000 (0:00:00.062) 0:04:53.538 ******* 2021-11-19T08:30:16.707 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-11-19T08:30:16.709 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:16.730 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:16 +0000 (0:00:00.051) 0:04:53.589 ******* 2021-11-19T08:30:16.759 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-19T08:30:16.761 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:16.784 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:16 +0000 (0:00:00.053) 0:04:53.643 ******* 2021-11-19T08:30:16.813 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:16.836 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:16 +0000 (0:00:00.051) 0:04:53.694 ******* 2021-11-19T08:30:16.868 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-19T08:30:16.870 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:16.892 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:16 +0000 (0:00:00.056) 0:04:53.751 ******* 2021-11-19T08:30:16.919 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-19T08:30:16.921 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:16.941 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:16 +0000 (0:00:00.049) 0:04:53.800 ******* 2021-11-19T08:30:16.969 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-19T08:30:16.971 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:16.992 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:16 +0000 (0:00:00.050) 0:04:53.850 ******* 2021-11-19T08:30:17.019 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-19T08:30:17.021 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.042 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.049) 0:04:53.900 ******* 2021-11-19T08:30:17.069 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.090 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.048) 0:04:53.949 ******* 2021-11-19T08:30:17.117 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-11-19T08:30:17.118 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.140 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.050) 0:04:53.999 ******* 2021-11-19T08:30:17.168 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-19T08:30:17.170 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.189 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.048) 0:04:54.048 ******* 2021-11-19T08:30:17.216 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-19T08:30:17.218 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.238 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.048) 0:04:54.097 ******* 2021-11-19T08:30:17.265 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-11-19T08:30:17.267 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.287 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.049) 0:04:54.146 ******* 2021-11-19T08:30:17.315 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-11-19T08:30:17.316 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.337 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.050) 0:04:54.196 ******* 2021-11-19T08:30:17.364 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-11-19T08:30:17.366 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.386 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.048) 0:04:54.245 ******* 2021-11-19T08:30:17.412 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-11-19T08:30:17.414 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.441 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.054) 0:04:54.299 ******* 2021-11-19T08:30:17.474 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-11-19T08:30:17.475 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.503 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.062) 0:04:54.362 ******* 2021-11-19T08:30:17.533 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-11-19T08:30:17.535 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.561 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.057) 0:04:54.420 ******* 2021-11-19T08:30:17.589 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-11-19T08:30:17.590 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.616 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.054) 0:04:54.475 ******* 2021-11-19T08:30:17.645 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-11-19T08:30:17.647 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.668 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.052) 0:04:54.527 ******* 2021-11-19T08:30:17.696 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.718 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.049) 0:04:54.577 ******* 2021-11-19T08:30:17.756 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:30:17.757 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-11-19T08:30:17.777 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.059) 0:04:54.636 ******* 2021-11-19T08:30:17.816 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-11-19T08:30:17.818 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.838 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-11-19T08:30:17.840 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.850 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.852 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.861 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.863 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:17.888 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:17 +0000 (0:00:00.110) 0:04:54.746 ******* 2021-11-19T08:30:18.141 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-11-19T08:30:18.143 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004983", "end": "2021-11-19 08:30:18.115639", "rc": 0, "start": "2021-11-19 08:30:18.110656", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock"]} 2021-11-19T08:30:18.176 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:18 +0000 (0:00:00.288) 0:04:55.035 ******* 2021-11-19T08:30:18.204 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-11-19T08:30:18.205 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:18.230 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:18 +0000 (0:00:00.053) 0:04:55.089 ******* 2021-11-19T08:30:18.259 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-11-19T08:30:18.261 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:18.284 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:18 +0000 (0:00:00.053) 0:04:55.142 ******* 2021-11-19T08:30:18.312 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-11-19T08:30:18.313 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:18.334 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:18 +0000 (0:00:00.050) 0:04:55.193 ******* 2021-11-19T08:30:18.361 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-11-19T08:30:18.363 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:18.383 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:18 +0000 (0:00:00.049) 0:04:55.242 ******* 2021-11-19T08:30:20.378 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-11-19T08:30:20.380 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:30:20.402 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:20 +0000 (0:00:02.018) 0:04:57.261 ******* 2021-11-19T08:30:20.440 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-19T08:30:20.442 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140123136, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227084991, 'block_used': 13010117, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227084991, "block_size": 4096, "block_total": 240095108, "block_used": 13010117, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140123136, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}} 2021-11-19T08:30:20.465 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:20 +0000 (0:00:00.062) 0:04:57.324 ******* 2021-11-19T08:30:20.660 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-11-19T08:30:20.662 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005267", "end": "2021-11-19 08:30:20.649964", "rc": 0, "start": "2021-11-19 08:30:20.644697", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:20.684 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:20 +0000 (0:00:00.218) 0:04:57.543 ******* 2021-11-19T08:30:20.721 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-11-19T08:30:20.723 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140123136, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227084991, 'block_used': 13010117, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227084991, "block_size": 4096, "block_total": 240095108, "block_used": 13010117, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140123136, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:20.744 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:20 +0000 (0:00:00.060) 0:04:57.603 ******* 2021-11-19T08:30:20.772 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-11-19T08:30:20.794 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:20 +0000 (0:00:00.049) 0:04:57.652 ******* 2021-11-19T08:30:21.125 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: systemd-pam-239-51.el8.x86_64", "Installed: systemd-239-51.el8.x86_64", "Installed: systemd-libs-239-51.el8.x86_64", "Installed: systemd-udev-239-51.el8.x86_64", "Removed: systemd-239-45.el8.x86_64", "Removed: systemd-libs-239-45.el8.x86_64", "Removed: systemd-pam-239-45.el8.x86_64", "Removed: systemd-udev-239-45.el8.x86_64"]} 2021-11-19T08:30:21.148 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:21 +0000 (0:00:00.354) 0:04:58.007 ******* 2021-11-19T08:30:21.468 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-11-19T08:30:21.469 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service dbus.socket system.slice sysinit.target basic.target dbus.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "iptables.service nftables.service ebtables.service shutdown.target ip6tables.service ipset.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate" 2021-11-19T08:30:21.471 INFO:teuthology.task.ansible.out:: "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": 2021-11-19T08:30:21.473 INFO:teuthology.task.ansible.out:"0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice dbus.socket sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "nul 2021-11-19T08:30:21.474 INFO:teuthology.task.ansible.out:l", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-11-19T08:30:21.492 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:21 +0000 (0:00:00.343) 0:04:58.351 ******* 2021-11-19T08:30:21.790 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-11-19T08:30:21.792 INFO:teuthology.task.ansible.out:fatal: [smithi023.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-11-19T08:30:21.813 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:21 +0000 (0:00:00.320) 0:04:58.671 ******* 2021-11-19T08:30:22.101 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-11-19T08:30:22.103 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-11-19T08:30:22.123 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.310) 0:04:58.982 ******* 2021-11-19T08:30:22.151 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-11-19T08:30:22.153 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.174 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.051) 0:04:59.033 ******* 2021-11-19T08:30:22.212 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-11-19T08:30:22.214 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.235 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.060) 0:04:59.093 ******* 2021-11-19T08:30:22.262 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-11-19T08:30:22.263 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.284 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.049) 0:04:59.143 ******* 2021-11-19T08:30:22.312 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-11-19T08:30:22.314 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.335 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.051) 0:04:59.194 ******* 2021-11-19T08:30:22.364 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-11-19T08:30:22.366 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.389 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.053) 0:04:59.247 ******* 2021-11-19T08:30:22.426 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-11-19T08:30:22.428 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.448 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.059) 0:04:59.307 ******* 2021-11-19T08:30:22.476 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-11-19T08:30:22.497 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.048) 0:04:59.356 ******* 2021-11-19T08:30:22.524 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-11-19T08:30:22.526 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.551 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.053) 0:04:59.409 ******* 2021-11-19T08:30:22.578 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-11-19T08:30:22.580 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.600 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.049) 0:04:59.459 ******* 2021-11-19T08:30:22.640 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-19T08:30:22.641 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvtzgGdUVHD7ojvkwfML5oSDmhUCyNsVcP'], 'uuids': ['123585d8-93ba-4238-9f45-201e50793b77'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.630395", "end": "2021-11-19 08:30:22.624846", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvtzgGdUVHD7ojvkwfML5oSDmhUCyNsVcP"], "labels": [], "masters": [], "uuids": ["123585d8-93ba-4238-9f45-201e50793b77"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:20.994451", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} TASK [testnode : Install apt keys] ********* 2021-11-19T08:30:22.643 INFO:teuthology.task.ansible.out:************************************ skipping: [smithi023.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.652 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.675 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.074) 0:04:59.533 ******* 2021-11-19T08:30:22.701 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-11-19T08:30:22.703 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.723 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.048) 0:04:59.582 ******* 2021-11-19T08:30:22.760 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-11-19T08:30:22.782 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.059) 0:04:59.641 ******* 2021-11-19T08:30:22.809 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-11-19T08:30:22.811 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.831 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.048) 0:04:59.690 ******* 2021-11-19T08:30:22.858 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-11-19T08:30:22.860 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.881 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.049) 0:04:59.739 ******* 2021-11-19T08:30:22.910 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-11-19T08:30:22.912 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.933 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.052) 0:04:59.791 ******* 2021-11-19T08:30:22.960 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-19T08:30:22.962 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:22.982 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:22 +0000 (0:00:00.049) 0:04:59.841 ******* 2021-11-19T08:30:23.020 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-11-19T08:30:23.022 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.042 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.059) 0:04:59.901 ******* 2021-11-19T08:30:23.071 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-19T08:30:23.072 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.096 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.053) 0:04:59.955 ******* 2021-11-19T08:30:23.124 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-11-19T08:30:23.126 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.148 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.052) 0:05:00.007 ******* 2021-11-19T08:30:23.175 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-11-19T08:30:23.177 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.198 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.049) 0:05:00.056 ******* 2021-11-19T08:30:23.225 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-11-19T08:30:23.226 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.247 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.049) 0:05:00.106 ******* 2021-11-19T08:30:23.274 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.296 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.049) 0:05:00.155 ******* 2021-11-19T08:30:23.323 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-11-19T08:30:23.325 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.345 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.049) 0:05:00.204 ******* 2021-11-19T08:30:23.383 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-11-19T08:30:23.385 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.405 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.059) 0:05:00.263 ******* 2021-11-19T08:30:23.432 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-11-19T08:30:23.434 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.454 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.049) 0:05:00.313 ******* 2021-11-19T08:30:23.483 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-11-19T08:30:23.485 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.505 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.051) 0:05:00.364 ******* 2021-11-19T08:30:23.533 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-11-19T08:30:23.534 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.554 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.048) 0:05:00.413 ******* 2021-11-19T08:30:23.581 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-11-19T08:30:23.582 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.602 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.048) 0:05:00.461 ******* 2021-11-19T08:30:23.629 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-11-19T08:30:23.631 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.651 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.048) 0:05:00.510 ******* 2021-11-19T08:30:23.678 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-11-19T08:30:23.680 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.700 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.048) 0:05:00.559 ******* 2021-11-19T08:30:23.727 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-11-19T08:30:23.729 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.751 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.051) 0:05:00.610 ******* 2021-11-19T08:30:23.778 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-11-19T08:30:23.779 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.801 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.050) 0:05:00.660 ******* 2021-11-19T08:30:23.829 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-11-19T08:30:23.830 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.851 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.049) 0:05:00.710 ******* 2021-11-19T08:30:23.888 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-11-19T08:30:23.890 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.911 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.060) 0:05:00.770 ******* 2021-11-19T08:30:23.919 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-19T08:30:23.920 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-S81oJG-b7i3-wlky-DLEN-E2BQ-nbre-cnJRa5', 'nvme-INTEL_SSDPEDMD400G4_CVFT62420002400BGN', 'nvme-nvme.8086-43564654363234323030303234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.064887", "end": "2021-11-19 08:30:23.881067", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-S81oJG-b7i3-wlky-DLEN-E2BQ-nbre-cnJRa5", "nvme-INTEL_SSDPEDMD400G4_CVFT62420002400BGN", "nvme-nvme.8086-43564654363234323030303234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:22.816180", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-11-19T08:30:23.921 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-19T08:30:23.942 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:23.965 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:23 +0000 (0:00:00.053) 0:05:00.823 ******* 2021-11-19T08:30:23.992 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:30:23.994 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:24.014 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:24 +0000 (0:00:00.049) 0:05:00.873 ******* 2021-11-19T08:30:24.041 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-11-19T08:30:24.043 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:24.063 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:24 +0000 (0:00:00.049) 0:05:00.922 ******* 2021-11-19T08:30:24.090 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-11-19T08:30:24.092 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:24.112 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:24 +0000 (0:00:00.048) 0:05:00.971 ******* 2021-11-19T08:30:24.139 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-11-19T08:30:24.141 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:24.161 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:24 +0000 (0:00:00.048) 0:05:01.020 ******* 2021-11-19T08:30:24.187 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:24.209 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:24 +0000 (0:00:00.048) 0:05:01.068 ******* 2021-11-19T08:30:24.435 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-19T08:30:24.436 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-11-19T08:30:24.457 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:24 +0000 (0:00:00.247) 0:05:01.316 ******* 2021-11-19T08:30:24.682 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-19T08:30:24.684 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-19T08:30:24.703 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:24 +0000 (0:00:00.246) 0:05:01.562 ******* 2021-11-19T08:30:24.911 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-19T08:30:24.913 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-11-19T08:30:24.937 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:24 +0000 (0:00:00.233) 0:05:01.795 ******* 2021-11-19T08:30:24.964 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-19T08:30:24.966 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:24.985 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:24 +0000 (0:00:00.048) 0:05:01.844 ******* 2021-11-19T08:30:25.013 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-19T08:30:25.014 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:25.035 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.049) 0:05:01.894 ******* 2021-11-19T08:30:25.062 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:25.085 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.049) 0:05:01.943 ******* 2021-11-19T08:30:25.112 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-11-19T08:30:25.114 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:25.134 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.049) 0:05:01.993 ******* 2021-11-19T08:30:25.161 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-19T08:30:25.163 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDv2ZL2TyVQTacUgezzXP29iG2A9fifG2hc'], 'uuids': ['0d290037-26bd-49d0-a2d9-bb9460aebb7e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.065872", "end": "2021-11-19 08:30:25.149740", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDv2ZL2TyVQTacUgezzXP29iG2A9fifG2hc"], "labels": [], "masters": [], "uuids": ["0d290037-26bd-49d0-a2d9-bb9460aebb7e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:24.083868", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot o 2021-11-19T08:30:25.165 INFO:teuthology.task.ansible.out:r after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-19T08:30:25.186 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.052) 0:05:02.045 ******* 2021-11-19T08:30:25.215 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-19T08:30:25.217 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:25.237 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.051) 0:05:02.096 ******* 2021-11-19T08:30:25.266 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:25.289 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.051) 0:05:02.148 ******* 2021-11-19T08:30:25.318 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-11-19T08:30:25.319 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:25.341 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.051) 0:05:02.200 ******* 2021-11-19T08:30:25.368 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-11-19T08:30:25.370 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:25.391 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.049) 0:05:02.249 ******* 2021-11-19T08:30:25.418 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-11-19T08:30:25.420 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:25.440 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.049) 0:05:02.299 ******* 2021-11-19T08:30:25.468 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-11-19T08:30:25.470 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:25.494 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.053) 0:05:02.352 ******* 2021-11-19T08:30:25.523 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-11-19T08:30:25.526 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:25.546 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.052) 0:05:02.405 ******* 2021-11-19T08:30:25.575 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-11-19T08:30:25.577 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:25.598 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.051) 0:05:02.457 ******* 2021-11-19T08:30:25.626 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-11-19T08:30:25.628 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:25.654 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.055) 0:05:02.512 ******* 2021-11-19T08:30:25.692 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-11-19T08:30:25.694 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:25.719 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.064) 0:05:02.577 ******* 2021-11-19T08:30:25.747 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-11-19T08:30:25.749 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:25.952 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.233) 0:05:02.811 ******* 2021-11-19T08:30:25.980 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-11-19T08:30:25.982 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.002 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:25 +0000 (0:00:00.049) 0:05:02.861 ******* 2021-11-19T08:30:26.029 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-11-19T08:30:26.031 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.052 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:26 +0000 (0:00:00.049) 0:05:02.910 ******* 2021-11-19T08:30:26.090 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-11-19T08:30:26.092 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.113 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:26 +0000 (0:00:00.060) 0:05:02.971 ******* 2021-11-19T08:30:26.140 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.162 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:26 +0000 (0:00:00.049) 0:05:03.021 ******* 2021-11-19T08:30:26.190 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.212 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:26 +0000 (0:00:00.050) 0:05:03.071 ******* 2021-11-19T08:30:26.240 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-11-19T08:30:26.241 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.262 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:26 +0000 (0:00:00.049) 0:05:03.121 ******* 2021-11-19T08:30:26.300 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:30:26.301 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-11-19T08:30:26.322 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:26 +0000 (0:00:00.060) 0:05:03.181 ******* 2021-11-19T08:30:26.372 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-11-19T08:30:26.374 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.383 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-11-19T08:30:26.384 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.395 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.396 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.406 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.408 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.433 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:26 +0000 (0:00:00.110) 0:05:03.292 ******* 2021-11-19T08:30:26.440 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-19T08:30:26.442 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvAq7Q1IxEnAHVysthzLWcsAnmaaJbGAps'], 'uuids': ['046e8b89-ebf6-4aaa-93f4-41c6c2d19175'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.085383", "end": "2021-11-19 08:30:26.419687", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvAq7Q1IxEnAHVysthzLWcsAnmaaJbGAps"], "labels": [], "masters": [], "uuids": ["046e8b89-ebf6-4aaa-93f4-41c6c2d19175"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:25.334304", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-19T08:30:26.630 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-11-19T08:30:26.632 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003336", "end": "2021-11-19 08:30:26.603041", "rc": 0, "start": "2021-11-19 08:30:26.599705", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/virtualenv/bin/teuthology-lock"]} 2021-11-19T08:30:26.654 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:26 +0000 (0:00:00.221) 0:05:03.513 ******* 2021-11-19T08:30:26.692 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-11-19T08:30:26.694 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.716 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:26 +0000 (0:00:00.062) 0:05:03.575 ******* 2021-11-19T08:30:26.744 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-11-19T08:30:26.745 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.766 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:26 +0000 (0:00:00.049) 0:05:03.624 ******* 2021-11-19T08:30:26.792 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-11-19T08:30:26.794 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.817 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:26 +0000 (0:00:00.051) 0:05:03.676 ******* 2021-11-19T08:30:26.844 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-11-19T08:30:26.845 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:26.866 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:26 +0000 (0:00:00.049) 0:05:03.725 ******* 2021-11-19T08:30:27.687 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-19T08:30:27.689 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvNneZoZ4KxrIn2gqfNlzEeB0KR3y0w1aj'], 'uuids': ['efab67fb-f06c-43dd-a896-870215a43699'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.067392", "end": "2021-11-19 08:30:27.674129", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvNneZoZ4KxrIn2gqfNlzEeB0KR3y0w1aj"], "labels": [], "masters": [], "uuids": ["efab67fb-f06c-43dd-a896-870215a43699"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:26.606737", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-19T08:30:27.700 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNGV', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNGV', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNGV', 'scsi-35000c50092941659', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XNGV', 'wwn-0x5000c50092941659'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XNGV', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50092941659', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNGV-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNGV-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNGV-part1', 'scsi-35000c50092941659-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XNGV-part1', 'wwn-0x5000c50092941659-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNGV", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNGV", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNGV", "scsi-35000c50092941659", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XNGV", "wwn-0x5000c50092941659"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNGV-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNGV-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-19T08:30:27.702 INFO:teuthology.task.ansible.out:Z1W5XNGV-part1", "scsi-35000c50092941659-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XNGV-part1", "wwn-0x5000c50092941659-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNGV", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092941659"}}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:28.985 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvofsBX77FJJkOH2dyk1nWeafXNpMuwD94'], 'uuids': ['ff794e36-6008-437d-a9aa-c26d4b90e9bf'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.097467", "end": "2021-11-19 08:30:28.971782", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvofsBX77FJJkOH2dyk1nWeafXNpMuwD94"], "labels": [], "masters": [], "uuids": ["ff794e36-6008-437d-a9aa-c26d4b90e9bf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:27.874315", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-19T08:30:29.016 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:29 +0000 (0:00:02.149) 0:05:05.875 ******* 2021-11-19T08:30:29.034 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-11-19T08:30:29.036 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:30:29.057 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:29 +0000 (0:00:00.041) 0:05:05.916 ******* 2021-11-19T08:30:29.105 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-19T08:30:29.107 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140053504, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227084974, 'block_used': 13010134, 'inode_total': 61054976, 'inode_available': 60984950, 'inode_used': 70026, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227084974, "block_size": 4096, "block_total": 240095108, "block_used": 13010134, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984950, "inode_total": 61054976, "inode_used": 70026, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140053504, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}} 2021-11-19T08:30:29.128 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:29 +0000 (0:00:00.071) 0:05:05.987 ******* 2021-11-19T08:30:29.332 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-11-19T08:30:29.334 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005482", "end": "2021-11-19 08:30:29.320258", "rc": 0, "start": "2021-11-19 08:30:29.314776", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:29.356 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:29 +0000 (0:00:00.227) 0:05:06.215 ******* 2021-11-19T08:30:29.393 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-19T08:30:29.395 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvtzgGdUVHD7ojvkwfML5oSDmhUCyNsVcP'], 'uuids': ['123585d8-93ba-4238-9f45-201e50793b77'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.152008", "end": "2021-11-19 08:30:29.380676", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvtzgGdUVHD7ojvkwfML5oSDmhUCyNsVcP"], "labels": [], "masters": [], "uuids": ["123585d8-93ba-4238-9f45-201e50793b77"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:29.228668", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149234 s, 773 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149234 s, 773 MB/s"], "stdout": "", "stdout_lines": []} TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi023.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140053504, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227084974, 'block_used': 13010134, 'inode_total': 61054976, 2021-11-19T08:30:29.397 INFO:teuthology.task.ansible.out:'inode_available': 60984950, 'inode_used': 70026, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227084974, "block_size": 4096, "block_total": 240095108, "block_used": 13010134, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984950, "inode_total": 61054976, "inode_used": 70026, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140053504, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:29.418 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:29 +0000 (0:00:00.062) 0:05:06.277 ******* 2021-11-19T08:30:29.445 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-11-19T08:30:29.466 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:29 +0000 (0:00:00.048) 0:05:06.325 ******* 2021-11-19T08:30:29.611 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-19T08:30:29.611 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-S81oJG-b7i3-wlky-DLEN-E2BQ-nbre-cnJRa5', 'nvme-INTEL_SSDPEDMD400G4_CVFT62420002400BGN', 'nvme-nvme.8086-43564654363234323030303234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039765", "end": "2021-11-19 08:30:29.593469", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-S81oJG-b7i3-wlky-DLEN-E2BQ-nbre-cnJRa5", "nvme-INTEL_SSDPEDMD400G4_CVFT62420002400BGN", "nvme-nvme.8086-43564654363234323030303234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:29.553704", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0371095 s, 3.1 GB/s", "std 2021-11-19T08:30:29.613 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0371095 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:29.928 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDv2ZL2TyVQTacUgezzXP29iG2A9fifG2hc'], 'uuids': ['0d290037-26bd-49d0-a2d9-bb9460aebb7e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.142312", "end": "2021-11-19 08:30:29.913645", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDv2ZL2TyVQTacUgezzXP29iG2A9fifG2hc"], "labels": [], "masters": [], "uuids": ["0d290037-26bd-49d0-a2d9-bb9460aebb7e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:29.771333", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139629 s, 826 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139629 s, 826 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:30.244 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvAq7Q1IxEnAHVysthzLWcsAnmaaJbGAps'], 'uuids': ['046e8b89-ebf6-4aaa-93f4-41c6c2d19175'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.143078", "end": "2021-11-19 08:30:30.228621", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvAq7Q1IxEnAHVysthzLWcsAnmaaJbGAps"], "labels": [], "masters": [], "uuids": ["046e8b89-ebf6-4aaa-93f4-41c6c2d19175"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:30.085543", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140452 s, 821 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140452 s, 821 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:30.556 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvNneZoZ4KxrIn2gqfNlzEeB0KR3y0w1aj'], 'uuids': ['efab67fb-f06c-43dd-a896-870215a43699'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.148108", "end": "2021-11-19 08:30:30.550009", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvNneZoZ4KxrIn2gqfNlzEeB0KR3y0w1aj"], "labels": [], "masters": [], "uuids": ["efab67fb-f06c-43dd-a896-870215a43699"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:30.401901", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145438 s, 793 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145438 s, 793 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:30.568 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNGV', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNGV', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNGV', 'scsi-35000c50092941659', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XNGV', 'wwn-0x5000c50092941659'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XNGV', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50092941659', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNGV-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNGV-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNGV-part1', 'scsi-35000c50092941659-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XNGV-part1', 'wwn-0x5000c50092941659-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNGV", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNGV", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNGV", "scsi-35000c50092941659", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XNGV", "wwn-0x5000c50092941659"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNGV-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNGV-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-11-19T08:30:30.570 INFO:teuthology.task.ansible.out:Z1W5XNGV-part1", "scsi-35000c50092941659-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XNGV-part1", "wwn-0x5000c50092941659-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNGV", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092941659"}}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:30.885 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvofsBX77FJJkOH2dyk1nWeafXNpMuwD94'], 'uuids': ['ff794e36-6008-437d-a9aa-c26d4b90e9bf'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.142775", "end": "2021-11-19 08:30:30.872121", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-fnjhI1t4IZka3fA04wHWADZcBE5FBpDvofsBX77FJJkOH2dyk1nWeafXNpMuwD94"], "labels": [], "masters": [], "uuids": ["ff794e36-6008-437d-a9aa-c26d4b90e9bf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:30.729346", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140064 s, 824 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140064 s, 824 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:30.915 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:30 +0000 (0:00:01.448) 0:05:07.774 ******* 2021-11-19T08:30:31.171 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-11-19T08:30:31.173 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.053171", "end": "2021-11-19 08:30:31.159625", "rc": 0, "start": "2021-11-19 08:30:31.106454", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:31.194 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:31 +0000 (0:00:00.278) 0:05:08.053 ******* 2021-11-19T08:30:31.263 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-19T08:30:31.265 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitIhQkhX79lWNFe7MQTxECUaiy0eBzsVg0'], 'uuids': ['5847f2ae-67d4-44cd-a323-0b0d8c463d9a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.565935", "end": "2021-11-19 08:30:31.249163", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitIhQkhX79lWNFe7MQTxECUaiy0eBzsVg0"], "labels": [], "masters": [], "uuids": ["5847f2ae-67d4-44cd-a323-0b0d8c463d9a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:29.683228", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-19T08:30:31.425 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-11-19T08:30:31.427 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.040152", "end": "2021-11-19 08:30:31.413703", "rc": 0, "start": "2021-11-19 08:30:31.373551", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:31.447 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:31 +0000 (0:00:00.253) 0:05:08.306 ******* 2021-11-19T08:30:31.474 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-11-19T08:30:31.499 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:31 +0000 (0:00:00.051) 0:05:08.357 ******* 2021-11-19T08:30:31.526 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-11-19T08:30:31.527 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:31.552 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:31 +0000 (0:00:00.052) 0:05:08.410 ******* 2021-11-19T08:30:31.592 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-11-19T08:30:31.594 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:31.630 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:31 +0000 (0:00:00.078) 0:05:08.489 ******* 2021-11-19T08:30:31.661 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-19T08:30:31.663 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:31.687 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:31 +0000 (0:00:00.057) 0:05:08.546 ******* 2021-11-19T08:30:31.716 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-11-19T08:30:31.717 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:31.739 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:31 +0000 (0:00:00.051) 0:05:08.597 ******* 2021-11-19T08:30:31.936 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-11-19T08:30:31.938 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-19T08:30:31.959 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:31 +0000 (0:00:00.220) 0:05:08.818 ******* 2021-11-19T08:30:32.007 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-19T08:30:32.008 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140123136, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227084991, 'block_used': 13010117, 'inode_total': 61054976, 'inode_available': 60984946, 'inode_used': 70030, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227084991, "block_size": 4096, "block_total": 240095108, "block_used": 13010117, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984946, "inode_total": 61054976, "inode_used": 70030, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140123136, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:32.031 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:32 +0000 (0:00:00.072) 0:05:08.890 ******* 2021-11-19T08:30:32.058 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-11-19T08:30:32.060 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:32.080 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:32 +0000 (0:00:00.048) 0:05:08.939 ******* 2021-11-19T08:30:32.117 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:30:32.119 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:32.139 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:32 +0000 (0:00:00.058) 0:05:08.998 ******* 2021-11-19T08:30:32.165 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:30:32.167 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:32.188 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:32 +0000 (0:00:00.048) 0:05:09.046 ******* 2021-11-19T08:30:32.225 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-11-19T08:30:32.226 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:32.246 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:32 +0000 (0:00:00.058) 0:05:09.105 ******* 2021-11-19T08:30:32.283 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-11-19T08:30:32.285 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:32.306 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:32 +0000 (0:00:00.059) 0:05:09.165 ******* 2021-11-19T08:30:32.343 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-11-19T08:30:32.345 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:32.365 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:32 +0000 (0:00:00.059) 0:05:09.224 ******* 2021-11-19T08:30:32.529 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-19T08:30:32.531 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-rtpXIq-3ppk-5neQ-m1cV-rVBA-ltwn-WRg9dj', 'nvme-INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN', 'nvme-eui.0100000001000000e4d25c000034df14'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDKE040T7', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '7814037168', 'sectorsize': '512', 'size': '3.64 TB', 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.063788", "end": "2021-11-19 08:30:32.510880", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]", "links": {"ids": ["lvm-pv-uuid-rtpXIq-3ppk-5neQ-m1cV-rVBA-ltwn-WRg9dj", "nvme-INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN", "nvme-eui.0100000001000000e4d25c000034df14"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDKE040T7", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "7814037168", "sectorsize": "512", "size": "3.64 TB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:31.447092", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_li 2021-11-19T08:30:32.532 INFO:teuthology.task.ansible.out:nes": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-19T08:30:32.892 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-11-19T08:30:32.894 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-11-19T08:30:32.914 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:32 +0000 (0:00:00.548) 0:05:09.773 ******* 2021-11-19T08:30:33.379 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-19T08:30:33.733 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-19T08:30:33.775 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-19T08:30:33.777 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitfd4x6bymLTDABhNrTNqa6dXnpKTafieu'], 'uuids': ['e85216e7-6458-44ce-80ca-e59b8b3a7f02'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.052485", "end": "2021-11-19 08:30:33.756105", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitfd4x6bymLTDABhNrTNqa6dXnpKTafieu"], "labels": [], "masters": [], "uuids": ["e85216e7-6458-44ce-80ca-e59b8b3a7f02"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:32.703620", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-19T08:30:34.067 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-11-19T08:30:34.068 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-19T08:30:34.421 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-19T08:30:34.776 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-19T08:30:34.799 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:34 +0000 (0:00:01.884) 0:05:11.658 ******* 2021-11-19T08:30:34.992 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-19T08:30:34.994 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-11-19T08:30:35.015 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:35 +0000 (0:00:00.215) 0:05:11.873 ******* 2021-11-19T08:30:35.053 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-19T08:30:35.055 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitJ6PvvYSdqDUOAuWOCKsdAaqsX80v4UW4'], 'uuids': ['73cb67b4-1d55-4ec3-9702-5905a8a4393c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.083454", "end": "2021-11-19 08:30:35.039629", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitJ6PvvYSdqDUOAuWOCKsdAaqsX80v4UW4"], "labels": [], "masters": [], "uuids": ["73cb67b4-1d55-4ec3-9702-5905a8a4393c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:33.956175", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-19T08:30:35.223 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-11-19T08:30:35.224 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-19T08:30:35.401 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-19T08:30:35.579 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-19T08:30:35.756 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-19T08:30:35.767 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:35.791 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:35 +0000 (0:00:00.776) 0:05:12.650 ******* 2021-11-19T08:30:35.819 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:35.840 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:35 +0000 (0:00:00.048) 0:05:12.699 ******* 2021-11-19T08:30:35.867 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-11-19T08:30:35.869 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:35.889 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:35 +0000 (0:00:00.049) 0:05:12.748 ******* 2021-11-19T08:30:36.092 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-11-19T08:30:36.094 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-19T08:30:36.118 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:36 +0000 (0:00:00.228) 0:05:12.977 ******* 2021-11-19T08:30:36.155 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-11-19T08:30:36.157 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-11-19T08:30:36.180 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:36 +0000 (0:00:00.061) 0:05:13.039 ******* 2021-11-19T08:30:36.343 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-19T08:30:36.345 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitZ9KHaiuStSvqxKFDlEozVLJIZ26iqKPv'], 'uuids': ['4501cbd1-5b2c-4e74-b85d-0db74f87bbac'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.081255", "end": "2021-11-19 08:30:36.319473", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitZ9KHaiuStSvqxKFDlEozVLJIZ26iqKPv"], "labels": [], "masters": [], "uuids": ["4501cbd1-5b2c-4e74-b85d-0db74f87bbac"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:35.238218", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-19T08:30:36.345 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => 2021-11-19T08:30:36.347 INFO:teuthology.task.ansible.out:(item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPS3', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3', 'scsi-35000c500871a974c', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3', 'wwn-0x5000c500871a974c'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HPS3', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a974c', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPS3-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3-part1', 'scsi-35000c500871a974c-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3-part1', 'wwn-0x5000c500871a974c-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPS3", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3", "scsi-35000c500871a974c", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3", "wwn-0x5000c500871a974c"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPS3-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3-part1", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3-part1", "scsi-35000c500871a974c-part1 2021-11-19T08:30:36.349 INFO:teuthology.task.ansible.out:", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3-part1", "wwn-0x5000c500871a974c-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPS3", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a974c"}}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:36.554 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi134.front.sepia.ceph.com] => {"changed": true} 2021-11-19T08:30:36.578 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:36 +0000 (0:00:00.397) 0:05:13.437 ******* 2021-11-19T08:30:36.980 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-11-19T08:30:36.982 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-11-19T08:30:37.002 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:36 +0000 (0:00:00.424) 0:05:13.861 ******* 2021-11-19T08:30:37.029 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-11-19T08:30:37.031 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.051 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.048) 0:05:13.910 ******* 2021-11-19T08:30:37.078 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-11-19T08:30:37.080 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.100 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.049) 0:05:13.959 ******* 2021-11-19T08:30:37.127 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-11-19T08:30:37.129 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.149 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.048) 0:05:14.008 ******* 2021-11-19T08:30:37.176 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-11-19T08:30:37.178 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.198 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.048) 0:05:14.057 ******* 2021-11-19T08:30:37.225 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-11-19T08:30:37.227 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.247 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.049) 0:05:14.106 ******* 2021-11-19T08:30:37.274 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-11-19T08:30:37.275 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.296 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.048) 0:05:14.154 ******* 2021-11-19T08:30:37.322 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-11-19T08:30:37.324 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.344 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.048) 0:05:14.203 ******* 2021-11-19T08:30:37.371 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-11-19T08:30:37.373 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.396 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.051) 0:05:14.255 ******* 2021-11-19T08:30:37.423 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-11-19T08:30:37.425 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.446 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.049) 0:05:14.305 ******* 2021-11-19T08:30:37.473 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.495 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.049) 0:05:14.354 ******* 2021-11-19T08:30:37.523 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-11-19T08:30:37.525 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.545 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.049) 0:05:14.404 ******* 2021-11-19T08:30:37.572 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-11-19T08:30:37.573 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.596 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.050) 0:05:14.455 ******* 2021-11-19T08:30:37.633 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-11-19T08:30:37.635 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.655 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.059) 0:05:14.514 ******* 2021-11-19T08:30:37.661 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-11-19T08:30:37.662 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitrofpc2H862FLmo9co1PBpFwKdLYGypeE'], 'uuids': ['e0681676-9b14-4569-966d-29c89b12714d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.105471", "end": "2021-11-19 08:30:37.629860", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitrofpc2H862FLmo9co1PBpFwKdLYGypeE"], "labels": [], "masters": [], "uuids": ["e0681676-9b14-4569-966d-29c89b12714d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:36.524389", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-11-19T08:30:37.692 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.037) 0:05:14.551 ******* 2021-11-19T08:30:37.699 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-11-19T08:30:37.701 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.724 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.031) 0:05:14.583 ******* 2021-11-19T08:30:37.751 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-11-19T08:30:37.753 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:37.776 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:37 +0000 (0:00:00.052) 0:05:14.635 ******* 2021-11-19T08:30:38.109 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitIhQkhX79lWNFe7MQTxECUaiy0eBzsVg0'], 'uuids': ['5847f2ae-67d4-44cd-a323-0b0d8c463d9a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.187121", "end": "2021-11-19 08:30:38.098552", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitIhQkhX79lWNFe7MQTxECUaiy0eBzsVg0"], "labels": [], "masters": [], "uuids": ["5847f2ae-67d4-44cd-a323-0b0d8c463d9a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:37.911431", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.184243 s, 626 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.184243 s, 626 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:38.255 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-11-19T08:30:38.257 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-11-19T08:30:38.277 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:38 +0000 (0:00:00.500) 0:05:15.136 ******* 2021-11-19T08:30:38.338 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-rtpXIq-3ppk-5neQ-m1cV-rVBA-ltwn-WRg9dj', 'nvme-INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN', 'nvme-eui.0100000001000000e4d25c000034df14'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDKE040T7', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '7814037168', 'sectorsize': '512', 'size': '3.64 TB', 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.038996", "end": "2021-11-19 08:30:38.321914", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]", "links": {"ids": ["lvm-pv-uuid-rtpXIq-3ppk-5neQ-m1cV-rVBA-ltwn-WRg9dj", "nvme-INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN", "nvme-eui.0100000001000000e4d25c000034df14"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDKE040T7", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "7814037168", "sectorsize": "512", "size": "3.64 TB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:38.282918", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0361483 s, 3.2 GB/ 2021-11-19T08:30:38.340 INFO:teuthology.task.ansible.out:s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0361483 s, 3.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:38.579 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-11-19T08:30:38.579 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-11-19 08:24:26 UTC", "ActiveEnterTimestampMonotonic": "12124339", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target sysinit.target systemd-journald.socket systemd-tmpfiles-setup.service system.slice ntpd.service sntp.service ntpdate.service -.mount tmp.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-11-19 08:24:25 UTC", "AssertTimestampMonotonic": "10695447", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-11-19 08:24:25 UTC", "ConditionTimestampMonotonic": "10695404", "ConfigurationDirectoryMode": "0755", "Conflicts": "shut 2021-11-19T08:30:38.581 INFO:teuthology.task.ansible.out:down.target systemd-timesyncd.service ntpd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "877", "ExecMainStartTimestamp": "Fri 2021-11-19 08:24:26 UTC", "ExecMainStartTimestampMonotonic": "12064859", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-11-19 08:24:25 UTC", "InactiveExitTimestampMonotonic": "10801414", "InvocationID": "36770669693b4d15b7183929db85816b", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": 2021-11-19T08:30:38.583 INFO:teuthology.task.ansible.out:"infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "877", "MemoryAccounting": "yes", "MemoryCurrent": "1597440", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit 2021-11-19T08:30:38.584 INFO:teuthology.task.ansible.out:.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-11-19 08:24:26 UTC", "StateChangeTimestampMonotonic": "12124339", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-11-19 08:24:26 UTC", "WatchdogTimestampMonotonic": "12064879", "WatchdogUSec": "0"}} 2021-11-19T08:30:38.602 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:38 +0000 (0:00:00.324) 0:05:15.461 ******* 2021-11-19T08:30:38.629 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:30:38.631 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-11-19T08:30:38.652 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:38 +0000 (0:00:00.049) 0:05:15.511 ******* 2021-11-19T08:30:38.671 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-19T08:30:38.672 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitfd4x6bymLTDABhNrTNqa6dXnpKTafieu'], 'uuids': ['e85216e7-6458-44ce-80ca-e59b8b3a7f02'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.150784", "end": "2021-11-19 08:30:38.655425", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitfd4x6bymLTDABhNrTNqa6dXnpKTafieu"], "labels": [], "masters": [], "uuids": ["e85216e7-6458-44ce-80ca-e59b8b3a7f02"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:38.504641", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147993 s, 779 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147993 s, 779 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:38.704 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:30:38.705 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-11-19T08:30:38.733 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:38 +0000 (0:00:00.081) 0:05:15.592 ******* 2021-11-19T08:30:38.771 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-11-19T08:30:38.773 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:38.793 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:38 +0000 (0:00:00.060) 0:05:15.652 ******* 2021-11-19T08:30:38.820 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-11-19T08:30:38.822 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:38.842 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:38 +0000 (0:00:00.049) 0:05:15.701 ******* 2021-11-19T08:30:38.869 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-11-19T08:30:38.871 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:38.892 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:38 +0000 (0:00:00.048) 0:05:15.750 ******* 2021-11-19T08:30:39.003 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-19T08:30:39.004 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitJ6PvvYSdqDUOAuWOCKsdAaqsX80v4UW4'], 'uuids': ['73cb67b4-1d55-4ec3-9702-5905a8a4393c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.148252", "end": "2021-11-19 08:30:38.986972", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitJ6PvvYSdqDUOAuWOCKsdAaqsX80v4UW4"], "labels": [], "masters": [], "uuids": ["73cb67b4-1d55-4ec3-9702-5905a8a4393c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:38.838720", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145296 s, 794 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145296 s, 794 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:39.087 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-11-19T08:30:39.089 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-11-19T08:30:39.111 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:39 +0000 (0:00:00.219) 0:05:15.969 ******* 2021-11-19T08:30:39.326 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-19T08:30:39.328 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitZ9KHaiuStSvqxKFDlEozVLJIZ26iqKPv'], 'uuids': ['4501cbd1-5b2c-4e74-b85d-0db74f87bbac'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.144988", "end": "2021-11-19 08:30:39.311858", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitZ9KHaiuStSvqxKFDlEozVLJIZ26iqKPv"], "labels": [], "masters": [], "uuids": ["4501cbd1-5b2c-4e74-b85d-0db74f87bbac"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:39.166870", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142218 s, 811 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142218 s, 811 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:39.328 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPS3', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3', 'scsi-35000c500871a974c', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3', 'wwn-0x5000c500871a974c'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_ 2021-11-19T08:30:39.330 INFO:teuthology.task.ansible.out:address': None, 'sas_device_handle': None, 'serial': 'Z1W4HPS3', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a974c', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPS3-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3-part1', 'scsi-35000c500871a974c-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3-part1', 'wwn-0x5000c500871a974c-part1'], 'uuids': ['55348bda-8fc5-47b8-bbaf-957635529c89'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPS3", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3", "scsi-35000c500871a974c", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3", "wwn-0x5000c500871a974c"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPS3-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3-part1", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3-part1", "scsi-35000c500871a974c-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3-part1", "wwn-0x5000c500871a974c-part1"], "labels": [], "masters": [], "uuids": ["55348bda-8fc5-47b8-bbaf-957635529c89"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_ha 2021-11-19T08:30:39.332 INFO:teuthology.task.ansible.out:ndle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPS3", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a974c"}}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:39.587 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-11-19T08:30:39.589 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-11-19T08:30:39.612 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:39 +0000 (0:00:00.501) 0:05:16.471 ******* 2021-11-19T08:30:39.650 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-11-19T08:30:39.652 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitrofpc2H862FLmo9co1PBpFwKdLYGypeE'], 'uuids': ['e0681676-9b14-4569-966d-29c89b12714d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.144582", "end": "2021-11-19 08:30:39.642769", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-NzykGeYsRtbyuA8T0kbcaeBqEhFFMcitrofpc2H862FLmo9co1PBpFwKdLYGypeE"], "labels": [], "masters": [], "uuids": ["e0681676-9b14-4569-966d-29c89b12714d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-11-19 08:30:39.498187", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141807 s, 813 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141807 s, 813 MB/s"], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:39.691 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:39 +0000 (0:00:00.078) 0:05:16.549 ******* 2021-11-19T08:30:39.937 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-11-19T08:30:39.939 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.052850", "end": "2021-11-19 08:30:39.925492", "rc": 0, "start": "2021-11-19 08:30:39.872642", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:39.960 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:39 +0000 (0:00:00.268) 0:05:16.818 ******* 2021-11-19T08:30:40.207 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-11-19T08:30:40.209 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.057655", "end": "2021-11-19 08:30:40.198470", "rc": 0, "start": "2021-11-19 08:30:40.140815", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:40.229 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:40 +0000 (0:00:00.269) 0:05:17.088 ******* 2021-11-19T08:30:40.256 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-11-19T08:30:40.280 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:40 +0000 (0:00:00.051) 0:05:17.139 ******* 2021-11-19T08:30:40.318 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-11-19T08:30:40.320 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:40.343 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:40 +0000 (0:00:00.062) 0:05:17.202 ******* 2021-11-19T08:30:40.381 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-11-19T08:30:40.382 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:40.408 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:40 +0000 (0:00:00.064) 0:05:17.267 ******* 2021-11-19T08:30:40.436 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-19T08:30:40.438 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:40.462 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:40 +0000 (0:00:00.053) 0:05:17.320 ******* 2021-11-19T08:30:40.489 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-11-19T08:30:40.491 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:40.511 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:40 +0000 (0:00:00.049) 0:05:17.370 ******* 2021-11-19T08:30:40.707 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-11-19T08:30:40.709 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-11-19T08:30:40.730 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:40 +0000 (0:00:00.218) 0:05:17.589 ******* 2021-11-19T08:30:40.746 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-11-19T08:30:40.748 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-11-19T08:30:40.770 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:40 +0000 (0:00:00.040) 0:05:17.629 ******* 2021-11-19T08:30:40.777 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-11-19T08:30:40.777 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930140053504, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227084974, 'block_used': 13010134, 'inode_total': 61054976, 'inode_available': 60984950, 'inode_used': 70026, 'uuid': '55348bda-8fc5-47b8-bbaf-957635529c89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227084974, "block_size": 4096, "block_total": 240095108, "block_used": 13010134, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60984950, "inode_total": 61054976, "inode_used": 70026, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930140053504, "size_total": 983429562368, "uuid": "55348bda-8fc5-47b8-bbaf-957635529c89"}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:40.802 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:40 +0000 (0:00:00.031) 0:05:17.660 ******* 2021-11-19T08:30:40.809 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-11-19T08:30:40.811 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:40.838 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:40 +0000 (0:00:00.035) 0:05:17.696 ******* 2021-11-19T08:30:40.845 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-11-19T08:30:40.847 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:40.869 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:40 +0000 (0:00:00.031) 0:05:17.728 ******* 2021-11-19T08:30:40.876 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-11-19T08:30:40.877 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-11-19T08:30:40.900 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:40 +0000 (0:00:00.030) 0:05:17.759 ******* 2021-11-19T08:30:40.906 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:30:40.908 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:40.931 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:40 +0000 (0:00:00.030) 0:05:17.790 ******* 2021-11-19T08:30:40.958 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:30:40.960 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:40.981 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:40 +0000 (0:00:00.050) 0:05:17.840 ******* 2021-11-19T08:30:41.008 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-11-19T08:30:41.010 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.031 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:41 +0000 (0:00:00.049) 0:05:17.890 ******* 2021-11-19T08:30:41.059 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-11-19T08:30:41.061 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.082 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:41 +0000 (0:00:00.050) 0:05:17.941 ******* 2021-11-19T08:30:41.089 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-11-19T08:30:41.091 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-11-19T08:30:41.114 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:41 +0000 (0:00:00.032) 0:05:17.973 ******* 2021-11-19T08:30:41.121 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-11-19T08:30:41.123 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.145 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:41 +0000 (0:00:00.031) 0:05:18.004 ******* 2021-11-19T08:30:41.152 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-11-19T08:30:41.154 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.177 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:41 +0000 (0:00:00.031) 0:05:18.036 ******* 2021-11-19T08:30:41.205 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-11-19T08:30:41.207 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-11-19T08:30:41.228 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:41 +0000 (0:00:00.051) 0:05:18.087 ******* 2021-11-19T08:30:41.256 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-11-19T08:30:41.257 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.294 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:41 +0000 (0:00:00.066) 0:05:18.153 ******* 2021-11-19T08:30:41.482 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-11-19T08:30:41.483 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-11-19T08:30:41.504 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:41 +0000 (0:00:00.209) 0:05:18.363 ******* 2021-11-19T08:30:41.594 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-11-19T08:30:41.596 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-11-19T08:30:41.625 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:41 +0000 (0:00:00.121) 0:05:18.484 ******* 2021-11-19T08:30:41.696 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-11-19T08:30:41.698 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-11-19T08:30:41.721 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:41 +0000 (0:00:00.095) 0:05:18.580 ******* 2021-11-19T08:30:41.749 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-11-19T08:30:41.751 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.771 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:41 +0000 (0:00:00.050) 0:05:18.630 ******* 2021-11-19T08:30:41.810 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-11-19T08:30:41.812 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi134.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.822 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.824 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi134.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.846 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:41 +0000 (0:00:00.075) 0:05:18.705 ******* 2021-11-19T08:30:41.884 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-11-19T08:30:41.886 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.896 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.897 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.907 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.909 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.932 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:41 +0000 (0:00:00.085) 0:05:18.790 ******* 2021-11-19T08:30:41.959 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:41.987 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:41 +0000 (0:00:00.053) 0:05:18.843 ******* 2021-11-19T08:30:42.012 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-11-19T08:30:42.014 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:42.036 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:42 +0000 (0:00:00.050) 0:05:18.894 ******* 2021-11-19T08:30:42.043 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-11-19T08:30:42.044 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-19T08:30:42.066 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:42.090 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:42 +0000 (0:00:00.054) 0:05:18.949 ******* 2021-11-19T08:30:42.118 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-11-19T08:30:42.120 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:42.140 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:42 +0000 (0:00:00.050) 0:05:18.999 ******* 2021-11-19T08:30:42.169 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-11-19T08:30:42.171 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:42.194 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:42 +0000 (0:00:00.053) 0:05:19.053 ******* 2021-11-19T08:30:42.232 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-11-19T08:30:42.234 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:42.258 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:42 +0000 (0:00:00.063) 0:05:19.116 ******* 2021-11-19T08:30:42.286 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-11-19T08:30:42.288 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:42.312 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:42 +0000 (0:00:00.054) 0:05:19.171 ******* 2021-11-19T08:30:42.340 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-11-19T08:30:42.342 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:42.362 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:42 +0000 (0:00:00.050) 0:05:19.221 ******* 2021-11-19T08:30:42.371 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-11-19T08:30:42.372 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-19T08:30:42.582 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-11-19T08:30:42.585 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005021", "end": "2021-11-19 08:30:42.561427", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-11-19 08:30:42.556406", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:42.709 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-19T08:30:42.762 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-11-19T08:30:42.763 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003102", "end": "2021-11-19 08:30:42.743293", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-11-19 08:30:42.740191", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:42.934 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002880", "end": "2021-11-19 08:30:42.916729", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-11-19 08:30:42.913849", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:43.082 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-19T08:30:43.103 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-11-19T08:30:43.105 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003203", "end": "2021-11-19 08:30:43.088514", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-11-19 08:30:43.085311", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:43.128 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:43 +0000 (0:00:00.765) 0:05:19.986 ******* 2021-11-19T08:30:43.156 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-11-19T08:30:43.179 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:43 +0000 (0:00:00.050) 0:05:20.037 ******* 2021-11-19T08:30:43.207 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-11-19T08:30:43.209 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:43.431 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-11-19T08:30:43.454 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:43 +0000 (0:00:00.275) 0:05:20.313 ******* 2021-11-19T08:30:43.660 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-11-19T08:30:43.662 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-11-19T08:30:43.683 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:43 +0000 (0:00:00.228) 0:05:20.541 ******* 2021-11-19T08:30:43.897 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-11-19T08:30:43.899 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-19T08:30:44.088 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-19T08:30:44.267 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-19T08:30:44.444 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-11-19T08:30:44.456 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:44.481 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:44 +0000 (0:00:00.798) 0:05:21.339 ******* 2021-11-19T08:30:44.509 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-11-19T08:30:44.511 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:44.532 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:44 +0000 (0:00:00.051) 0:05:21.391 ******* 2021-11-19T08:30:44.562 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:44.589 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:44 +0000 (0:00:00.055) 0:05:21.446 ******* 2021-11-19T08:30:44.792 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-11-19T08:30:44.793 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-19T08:30:44.814 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:44 +0000 (0:00:00.226) 0:05:21.673 ******* 2021-11-19T08:30:44.851 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-11-19T08:30:44.853 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-11-19T08:30:44.875 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:44 +0000 (0:00:00.061) 0:05:21.734 ******* 2021-11-19T08:30:45.318 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-11-19T08:30:45.320 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true} 2021-11-19T08:30:45.341 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:45 +0000 (0:00:00.465) 0:05:22.200 ******* 2021-11-19T08:30:45.661 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-11-19T08:30:45.662 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-11-19T08:30:45.682 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:45 +0000 (0:00:00.341) 0:05:22.541 ******* 2021-11-19T08:30:45.710 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-11-19T08:30:45.712 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:45.733 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:45 +0000 (0:00:00.050) 0:05:22.592 ******* 2021-11-19T08:30:45.759 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-11-19T08:30:45.761 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:45.782 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:45 +0000 (0:00:00.048) 0:05:22.641 ******* 2021-11-19T08:30:45.809 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-11-19T08:30:45.810 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:45.831 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:45 +0000 (0:00:00.048) 0:05:22.690 ******* 2021-11-19T08:30:45.863 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-11-19T08:30:45.865 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:45.886 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:45 +0000 (0:00:00.054) 0:05:22.745 ******* 2021-11-19T08:30:45.912 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-11-19T08:30:45.914 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:45.934 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:45 +0000 (0:00:00.048) 0:05:22.793 ******* 2021-11-19T08:30:45.961 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-11-19T08:30:45.963 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:45.984 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:45 +0000 (0:00:00.049) 0:05:22.843 ******* 2021-11-19T08:30:46.011 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-11-19T08:30:46.013 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:46.033 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:46 +0000 (0:00:00.049) 0:05:22.892 ******* 2021-11-19T08:30:46.060 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-11-19T08:30:46.062 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:46.083 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:46 +0000 (0:00:00.050) 0:05:22.942 ******* 2021-11-19T08:30:46.111 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-11-19T08:30:46.113 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:46.137 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:46 +0000 (0:00:00.053) 0:05:22.995 ******* 2021-11-19T08:30:46.164 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:46.187 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:46 +0000 (0:00:00.050) 0:05:23.046 ******* 2021-11-19T08:30:46.215 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-11-19T08:30:46.216 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:46.245 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:46 +0000 (0:00:00.057) 0:05:23.104 ******* 2021-11-19T08:30:46.284 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:46.306 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:46 +0000 (0:00:00.060) 0:05:23.165 ******* 2021-11-19T08:30:46.333 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-11-19T08:30:46.335 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:46.358 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:46 +0000 (0:00:00.051) 0:05:23.216 ******* 2021-11-19T08:30:46.385 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-11-19T08:30:46.387 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:46.407 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:46 +0000 (0:00:00.049) 0:05:23.266 ******* 2021-11-19T08:30:46.436 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-11-19T08:30:46.438 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:46.459 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:46 +0000 (0:00:00.051) 0:05:23.318 ******* 2021-11-19T08:30:46.957 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-11-19T08:30:46.960 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-11-19T08:30:46.980 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:46 +0000 (0:00:00.521) 0:05:23.839 ******* 2021-11-19T08:30:47.303 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-11-19T08:30:47.304 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-11-19 08:24:56 UTC", "ActiveEnterTimestampMonotonic": "11964281", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target ntpd.service system.slice sysinit.target sntp.service -.mount ntpdate.service tmp.mount systemd-tmpfiles-setup.service systemd-journald.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-11-19 08:24:54 UTC", "AssertTimestampMonotonic": "10362258", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-11-19 08:24:54 UTC", "ConditionTimestampMonotonic": "10362211", "ConfigurationDirectoryMode": "0755", "Conflicts": "syst 2021-11-19T08:30:47.306 INFO:teuthology.task.ansible.out:emd-timesyncd.service shutdown.target ntpd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "865", "ExecMainStartTimestamp": "Fri 2021-11-19 08:24:56 UTC", "ExecMainStartTimestampMonotonic": "11875055", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-11-19 08:24:54 UTC", "InactiveExitTimestampMonotonic": "10403737", "InvocationID": "3f0bd58d3eb747f98cd0f8a9ec8525ec", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": 2021-11-19T08:30:47.307 INFO:teuthology.task.ansible.out:"infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127243", "LimitNPROCSoft": "127243", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127243", "LimitSIGPENDINGSoft": "127243", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "865", "MemoryAccounting": "yes", "MemoryCurrent": "1597440", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target 2021-11-19T08:30:47.309 INFO:teuthology.task.ansible.out:-.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-11-19 08:24:56 UTC", "StateChangeTimestampMonotonic": "11964281", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203588", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-11-19 08:24:56 UTC", "WatchdogTimestampMonotonic": "11875079", "WatchdogUSec": "0"}} 2021-11-19T08:30:47.328 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:47 +0000 (0:00:00.347) 0:05:24.187 ******* 2021-11-19T08:30:47.356 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:30:47.358 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-11-19T08:30:47.377 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:47 +0000 (0:00:00.049) 0:05:24.236 ******* 2021-11-19T08:30:47.415 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:30:47.418 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-11-19T08:30:47.438 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:47 +0000 (0:00:00.060) 0:05:24.296 ******* 2021-11-19T08:30:47.465 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:47.487 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:47 +0000 (0:00:00.049) 0:05:24.346 ******* 2021-11-19T08:30:47.514 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-11-19T08:30:47.516 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:47.537 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:47 +0000 (0:00:00.050) 0:05:24.396 ******* 2021-11-19T08:30:47.567 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-11-19T08:30:47.568 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:47.589 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:47 +0000 (0:00:00.051) 0:05:24.448 ******* 2021-11-19T08:30:47.804 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-11-19T08:30:47.826 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:47 +0000 (0:00:00.237) 0:05:24.685 ******* 2021-11-19T08:30:48.326 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-11-19T08:30:48.328 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-11-19T08:30:48.348 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:48 +0000 (0:00:00.521) 0:05:25.207 ******* 2021-11-19T08:30:49.412 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-11-19T08:30:49.414 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-11-19T08:30:49.434 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:49 +0000 (0:00:01.085) 0:05:26.293 ******* 2021-11-19T08:30:49.462 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-11-19T08:30:49.464 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:49.484 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:49 +0000 (0:00:00.050) 0:05:26.343 ******* 2021-11-19T08:30:49.514 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-11-19T08:30:49.515 INFO:teuthology.task.ansible.out: 2021-11-19T08:30:49.536 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:49 +0000 (0:00:00.051) 0:05:26.395 ******* 2021-11-19T08:30:49.722 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-11-19T08:30:49.745 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:49 +0000 (0:00:00.209) 0:05:26.604 ******* 2021-11-19T08:30:49.774 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-11-19T08:30:49.776 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:49.796 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:49 +0000 (0:00:00.050) 0:05:26.655 ******* 2021-11-19T08:30:49.833 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-11-19T08:30:49.834 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-11-19T08:30:49.854 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:49 +0000 (0:00:00.058) 0:05:26.713 ******* 2021-11-19T08:30:49.881 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-11-19T08:30:49.883 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:49.906 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:49 +0000 (0:00:00.051) 0:05:26.764 ******* 2021-11-19T08:30:50.113 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-11-19T08:30:50.115 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-11-19T08:30:50.136 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:50 +0000 (0:00:00.230) 0:05:26.995 ******* 2021-11-19T08:30:50.342 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-11-19T08:30:50.344 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-11-19T08:30:50.365 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:50 +0000 (0:00:00.228) 0:05:27.224 ******* 2021-11-19T08:30:50.394 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-11-19T08:30:50.396 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:50.417 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:50 +0000 (0:00:00.051) 0:05:27.276 ******* 2021-11-19T08:30:50.455 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-11-19T08:30:50.457 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:50.467 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:50.469 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2021-11-19T08:30:50.492 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:50 +0000 (0:00:00.075) 0:05:27.351 ******* 2021-11-19T08:30:50.530 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-11-19T08:30:50.532 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:50.543 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:50.545 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:50.578 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:50 +0000 (0:00:00.086) 0:05:27.437 ******* 2021-11-19T08:30:50.606 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-11-19T08:30:50.608 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:50.634 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:50 +0000 (0:00:00.055) 0:05:27.493 ******* 2021-11-19T08:30:50.662 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:50.664 INFO:teuthology.task.ansible.out: 2021-11-19T08:30:50.685 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:50 +0000 (0:00:00.050) 0:05:27.544 ******* 2021-11-19T08:30:50.713 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-11-19T08:30:50.714 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:50.738 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:50 +0000 (0:00:00.053) 0:05:27.597 ******* 2021-11-19T08:30:50.765 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-11-19T08:30:50.768 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:50.788 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:50 +0000 (0:00:00.050) 0:05:27.647 ******* 2021-11-19T08:30:50.816 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-11-19T08:30:50.818 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:50.844 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:50 +0000 (0:00:00.055) 0:05:27.702 ******* 2021-11-19T08:30:50.871 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-11-19T08:30:50.873 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:50.897 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:50 +0000 (0:00:00.053) 0:05:27.755 ******* 2021-11-19T08:30:50.925 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-11-19T08:30:50.926 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:50.949 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:50 +0000 (0:00:00.052) 0:05:27.808 ******* 2021-11-19T08:30:50.978 INFO:teuthology.task.ansible.out: 2021-11-19T08:30:50.980 INFO:teuthology.task.ansible.out:TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:51.000 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:50 +0000 (0:00:00.050) 0:05:27.859 ******* 2021-11-19T08:30:52.224 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-11-19T08:30:52.226 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:01.006580", "end": "2021-11-19 08:30:52.205052", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-11-19 08:30:51.198472", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:52.421 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003211", "end": "2021-11-19 08:30:52.402684", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-11-19 08:30:52.399473", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:52.598 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002879", "end": "2021-11-19 08:30:52.579747", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-11-19 08:30:52.576868", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:53.774 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:01.004455", "end": "2021-11-19 08:30:53.757923", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-11-19 08:30:52.753468", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:30:53.798 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:53 +0000 (0:00:02.797) 0:05:30.656 ******* 2021-11-19T08:30:53.825 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-11-19T08:30:53.846 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:53 +0000 (0:00:00.048) 0:05:30.705 ******* 2021-11-19T08:30:53.873 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-11-19T08:30:53.875 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:53.940 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-11-19T08:30:53.956 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:53 +0000 (0:00:00.110) 0:05:30.815 ******* 2021-11-19T08:30:53.977 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:53 +0000 (0:00:00.020) 0:05:30.836 ******* 2021-11-19T08:30:54.625 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi023.front.sepia.ceph.com] 2021-11-19T08:30:54.655 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] 2021-11-19T08:30:54.682 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:54 +0000 (0:00:00.704) 0:05:31.541 ******* 2021-11-19T08:30:54.723 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:54 +0000 (0:00:00.041) 0:05:31.582 ******* 2021-11-19T08:30:54.730 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-19T08:30:54.732 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:54.754 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:54 +0000 (0:00:00.030) 0:05:31.613 ******* 2021-11-19T08:30:54.761 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:30:54.785 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:54 +0000 (0:00:00.031) 0:05:31.644 ******* 2021-11-19T08:30:54.793 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-19T08:30:54.795 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-11-19T08:30:54.818 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:54 +0000 (0:00:00.032) 0:05:31.677 ******* 2021-11-19T08:30:54.826 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-11-19T08:30:54.850 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:54 +0000 (0:00:00.032) 0:05:31.709 ******* 2021-11-19T08:30:54.869 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-11-19T08:30:54.870 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-11-19T08:30:54.893 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:54 +0000 (0:00:00.042) 0:05:31.751 ******* 2021-11-19T08:30:54.900 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-11-19T08:30:54.925 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:30:54 +0000 (0:00:00.032) 0:05:31.784 ******* 2021-11-19T08:31:38.144 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-11-19T08:31:38.146 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: containers-common-2:1-6.module_el8.6.0+954+963caf36.noarch", "Installed: podman-catatonit-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Installed: container-selinux-2:2.170.0-1.module_el8.6.0+954+963caf36.noarch", "Installed: podman-docker-1:3.4.1-3.module_el8.6.0+954+963caf36.noarch", "Installed: podman-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Removed: container-selinux-2:2.159.0-1.module_el8.5.0+733+9bb5dffa.noarch", "Removed: containers-common-1:1.2.2-4.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-catatonit-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-docker-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.noarch"]} 2021-11-19T08:31:38.170 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:38 +0000 (0:00:43.244) 0:06:15.029 ******* 2021-11-19T08:31:38.211 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi134.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-11-19T08:31:38.238 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:38 +0000 (0:00:00.068) 0:06:15.097 ******* 2021-11-19T08:31:38.268 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-19T08:31:38.270 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:31:38.291 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:38 +0000 (0:00:00.052) 0:06:15.150 ******* 2021-11-19T08:31:38.518 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* changed: [smithi134.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-19T08:31:38.709 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-19T08:31:38.746 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:38 +0000 (0:00:00.455) 0:06:15.605 ******* 2021-11-19T08:31:39.280 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-11-19T08:31:39.282 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310698.816331-3531-251222393684877/source", "state": "file", "uid": 0} 2021-11-19T08:31:39.769 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310699.3187706-3531-121023588350540/source", "state": "file", "uid": 0} 2021-11-19T08:31:39.793 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:39 +0000 (0:00:01.046) 0:06:16.652 ******* 2021-11-19T08:31:39.916 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-11-19T08:31:39.918 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: containers-common-2:1-6.module_el8.6.0+954+963caf36.noarch", "Installed: podman-catatonit-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Installed: container-selinux-2:2.170.0-1.module_el8.6.0+954+963caf36.noarch", "Installed: podman-docker-1:3.4.1-3.module_el8.6.0+954+963caf36.noarch", "Installed: podman-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64", "Removed: container-selinux-2:2.159.0-1.module_el8.5.0+733+9bb5dffa.noarch", "Removed: containers-common-1:1.2.2-4.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-catatonit-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64", "Removed: podman-docker-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.noarch"]} 2021-11-19T08:31:39.940 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:39 +0000 (0:00:00.147) 0:06:16.799 ******* 2021-11-19T08:31:39.970 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-19T08:31:39.972 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-11-19T08:31:39.992 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:39 +0000 (0:00:00.051) 0:06:16.851 ******* 2021-11-19T08:31:40.021 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-11-19T08:31:40.024 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:31:40.045 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:40 +0000 (0:00:00.052) 0:06:16.904 ******* 2021-11-19T08:31:40.295 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-11-19T08:31:40.297 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-19T08:31:40.486 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-11-19T08:31:40.509 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:40 +0000 (0:00:00.464) 0:06:17.368 ******* 2021-11-19T08:31:41.055 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-11-19T08:31:41.057 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310700.58513-3775-41058880894294/source", "state": "file", "uid": 0} 2021-11-19T08:31:41.569 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1637310701.0946095-3775-168950731898781/source", "state": "file", "uid": 0} 2021-11-19T08:31:41.592 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:41 +0000 (0:00:01.082) 0:06:18.450 ******* 2021-11-19T08:31:41.818 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2021-11-19T08:31:41.820 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:31:41.842 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:41 +0000 (0:00:00.250) 0:06:18.701 ******* 2021-11-19T08:31:43.698 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-11-19T08:31:43.732 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:43 +0000 (0:00:01.889) 0:06:20.591 ******* 2021-11-19T08:31:45.030 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-11-19T08:31:45.032 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-o0mlfp7k-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-o0mlfp7k-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-11-19T08:31:45.034 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-11-19T08:31:45.054 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:45 +0000 (0:00:01.321) 0:06:21.913 ******* 2021-11-19T08:31:45.083 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-11-19T08:31:45.085 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:31:45.109 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:45 +0000 (0:00:00.055) 0:06:21.968 ******* 2021-11-19T08:31:45.148 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-11-19T08:31:45.149 INFO:teuthology.task.ansible.out:skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:31:45.170 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:45 +0000 (0:00:00.060) 0:06:22.029 ******* 2021-11-19T08:31:45.519 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-11-19T08:31:45.521 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.149465", "end": "2021-11-19 08:31:45.512754", "rc": 0, "start": "2021-11-19 08:31:45.363289", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:31:45.545 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:45 +0000 (0:00:00.375) 0:06:22.404 ******* 2021-11-19T08:31:45.585 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* skipping: [smithi134.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:31:46.999 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-11-19T08:31:47.001 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-6jxj3y_u-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-6jxj3y_u-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-11-19T08:31:47.003 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-11-19T08:31:47.021 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:47 +0000 (0:00:01.475) 0:06:23.880 ******* 2021-11-19T08:31:47.049 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-11-19T08:31:47.050 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:31:47.080 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:47 +0000 (0:00:00.058) 0:06:23.939 ******* 2021-11-19T08:31:47.107 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-11-19T08:31:47.110 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:31:47.129 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:47 +0000 (0:00:00.049) 0:06:23.988 ******* 2021-11-19T08:31:47.491 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-11-19T08:31:47.493 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.150176", "end": "2021-11-19 08:31:47.479970", "rc": 0, "start": "2021-11-19 08:31:47.329794", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-11-19T08:31:47.513 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:47 +0000 (0:00:00.383) 0:06:24.372 ******* 2021-11-19T08:31:47.542 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-11-19T08:31:47.610 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-11-19T08:31:47.612 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-11-19T08:31:47.613 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-11-19T08:31:47.614 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-11-19T08:31:47.627 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-11-19T08:31:47.628 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:47 +0000 (0:00:00.113) 0:06:24.486 ******* 2021-11-19T08:31:48.285 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] 2021-11-19T08:31:48.306 INFO:teuthology.task.ansible.out:ok: [smithi134.front.sepia.ceph.com] 2021-11-19T08:31:48.323 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** 2021-11-19T08:31:48.324 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:48 +0000 (0:00:00.695) 0:06:25.182 ******* 2021-11-19T08:31:48.528 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-19T08:31:48.540 INFO:teuthology.task.ansible.out:changed: [smithi134.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-11-19T08:31:48.542 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi023.front.sepia.ceph.com : ok=138 changed=51 unreachable=0 failed=0 skipped=181 rescued=0 ignored=3 smithi134.front.sepia.ceph.com : ok=138 changed=51 unreachable=0 failed=0 skipped=181 rescued=0 ignored=3 2021-11-19T08:31:48.544 INFO:teuthology.task.ansible.out:Friday 19 November 2021 08:31:48 +0000 (0:00:00.219) 0:06:25.401 ******* =============================================================================== Create all admin users with sudo access. ------------------------------- 60.85s container-host : Install container packages ---------------------------- 43.24s common : nrpe - Ensure SELinux policy is up to date -------------------- 33.17s users : Update authorized_keys using the keys repo --------------------- 28.44s Remove revoked users --------------------------------------------------- 14.29s testnode : Ensure ceph packages are not present. ------------------------ 9.90s common : nrpe - Remove SELinux policy package --------------------------- 9.09s common : nrpe - Load SELinux policy package ----------------------------- 8.32s testnode : Ensure ceph dependency packages are not present. ------------- 3.37s testnode : List any leftover Ceph artifacts from previous jobs ---------- 2.80s testnode : Install GPG keys --------------------------------------------- 2.47s testnode : Make sure rpm dependencies are installed --------------------- 2.15s testnode : Configure local mirrorlists ---------------------------------- 1.94s common : nrpe - Install SELinux dependencies ---------------------------- 1.91s users : Update authorized_keys for each user with literal keys ---------- 1.89s testnode : Install GPG -------------------------------------------------- 1.87s testnode : Remove packages ---------------------------------------------- 1.63s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.52s common : Make sure smartmontools is installed --------------------------- 1.49s Gathering Facts --------------------------------------------------------- 1.48s 2021-11-19T08:31:48.959 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi023.front.sepia.ceph.com'), Remote(name='ubuntu@smithi134.front.sepia.ceph.com')] 2021-11-19T08:31:48.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2021-11-19T08:31:48.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-19T08:31:49.034 DEBUG:teuthology.orchestra.run.smithi023:> true 2021-11-19T08:31:49.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2021-11-19T08:31:49.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-11-19T08:31:49.175 DEBUG:teuthology.orchestra.run.smithi134:> true 2021-11-19T08:31:49.243 INFO:teuthology.run_tasks:Running task clock... 2021-11-19T08:31:49.253 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-11-19T08:31:49.254 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-11-19T08:31:49.256 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-19T08:31:49.258 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-11-19T08:31:49.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-19T08:31:49.288 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-11-19T08:31:49.302 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-11-19T08:31:49.313 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-11-19T08:31:49.327 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-11-19T08:31:49.340 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: ntpd: command not found 2021-11-19T08:31:49.351 INFO:teuthology.orchestra.run.smithi023.stdout:506 Cannot talk to daemon 2021-11-19T08:31:49.358 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2021-11-19T08:31:49.365 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-11-19T08:31:49.368 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2021-11-19T08:31:49.379 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-11-19T08:31:49.381 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-11-19T08:31:49.394 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-11-19T08:31:49.448 INFO:teuthology.orchestra.run.smithi023.stderr:bash: ntpq: command not found 2021-11-19T08:31:49.451 INFO:teuthology.orchestra.run.smithi023.stdout:210 Number of sources = 4 2021-11-19T08:31:49.451 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-19T08:31:49.451 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2021-11-19T08:31:49.451 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-19T08:31:49.452 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-19T08:31:49.452 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-19T08:31:49.452 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-19T08:31:49.460 INFO:teuthology.orchestra.run.smithi134.stderr:bash: ntpq: command not found 2021-11-19T08:31:49.463 INFO:teuthology.orchestra.run.smithi134.stdout:210 Number of sources = 4 2021-11-19T08:31:49.463 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-19T08:31:49.463 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2021-11-19T08:31:49.464 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-19T08:31:49.464 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-19T08:31:49.464 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-19T08:31:49.464 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-11-19T08:31:49.468 INFO:teuthology.run_tasks:Running task pexec... 2021-11-19T08:31:49.481 INFO:teuthology.task.pexec:Executing custom commands... 2021-11-19T08:31:49.482 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi023.front.sepia.ceph.com 2021-11-19T08:31:49.482 DEBUG:teuthology.orchestra.run.smithi023:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-11-19T08:31:49.483 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi134.front.sepia.ceph.com 2021-11-19T08:31:49.483 DEBUG:teuthology.orchestra.run.smithi134:> TESTDIR=/home/ubuntu/cephtest bash -s 2021-11-19T08:31:50.069 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:06 ago on Fri 19 Nov 2021 08:29:44 AM UTC. 2021-11-19T08:31:50.083 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:57 ago on Fri 19 Nov 2021 08:29:53 AM UTC. 2021-11-19T08:31:51.509 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:31:51.510 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:31:51.510 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2021-11-19T08:31:51.510 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:31:51.510 INFO:teuthology.orchestra.run.smithi134.stdout:Resetting modules: 2021-11-19T08:31:51.511 INFO:teuthology.orchestra.run.smithi134.stdout: container-tools 2021-11-19T08:31:51.511 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:31:51.511 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:31:51.511 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:31:51.512 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:31:51.724 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:31:51.786 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:31:51.787 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:31:51.788 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2021-11-19T08:31:51.788 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:31:51.788 INFO:teuthology.orchestra.run.smithi023.stdout:Resetting modules: 2021-11-19T08:31:51.788 INFO:teuthology.orchestra.run.smithi023.stdout: container-tools 2021-11-19T08:31:51.788 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:31:51.789 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:31:51.789 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:31:51.789 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:31:52.017 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:31:52.333 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:08 ago on Fri 19 Nov 2021 08:29:44 AM UTC. 2021-11-19T08:31:52.683 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:59 ago on Fri 19 Nov 2021 08:29:53 AM UTC. 2021-11-19T08:31:53.818 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:31:53.820 INFO:teuthology.orchestra.run.smithi134.stdout:===================================================================================================== 2021-11-19T08:31:53.821 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2021-11-19T08:31:53.821 INFO:teuthology.orchestra.run.smithi134.stdout:===================================================================================================== 2021-11-19T08:31:53.821 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2021-11-19T08:31:53.822 INFO:teuthology.orchestra.run.smithi134.stdout: conmon x86_64 2:2.0.30-1.module_el8.6.0+944+d413f95e CentOS-AppStream 54 k 2021-11-19T08:31:53.822 INFO:teuthology.orchestra.run.smithi134.stdout: containernetworking-plugins x86_64 1.0.1-1.module_el8.6.0+944+d413f95e CentOS-AppStream 19 M 2021-11-19T08:31:53.822 INFO:teuthology.orchestra.run.smithi134.stdout: criu x86_64 3.15-3.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 518 k 2021-11-19T08:31:53.822 INFO:teuthology.orchestra.run.smithi134.stdout: fuse-overlayfs x86_64 1.7.1-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 73 k 2021-11-19T08:31:53.823 INFO:teuthology.orchestra.run.smithi134.stdout: libslirp x86_64 4.4.0-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 70 k 2021-11-19T08:31:53.823 INFO:teuthology.orchestra.run.smithi134.stdout: runc x86_64 1.0.2-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 3.1 M 2021-11-19T08:31:53.823 INFO:teuthology.orchestra.run.smithi134.stdout: slirp4netns x86_64 1.1.8-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 51 k 2021-11-19T08:31:53.823 INFO:teuthology.orchestra.run.smithi134.stdout:Installing group/module packages: 2021-11-19T08:31:53.824 INFO:teuthology.orchestra.run.smithi134.stdout: buildah x86_64 1:1.23.1-2.module_el8.6.0+954+963caf36 CentOS-AppStream 7.9 M 2021-11-19T08:31:53.824 INFO:teuthology.orchestra.run.smithi134.stdout: cockpit-podman noarch 35-1.module_el8.6.0+944+d413f95e CentOS-AppStream 457 k 2021-11-19T08:31:53.824 INFO:teuthology.orchestra.run.smithi134.stdout: crun x86_64 1.2-1.module_el8.6.0+954+963caf36 CentOS-AppStream 193 k 2021-11-19T08:31:53.825 INFO:teuthology.orchestra.run.smithi134.stdout: python3-podman noarch 3.2.1-4.module_el8.6.0+954+963caf36 CentOS-AppStream 148 k 2021-11-19T08:31:53.825 INFO:teuthology.orchestra.run.smithi134.stdout: skopeo x86_64 2:1.5.0-2.module_el8.6.0+954+963caf36 CentOS-AppStream 6.7 M 2021-11-19T08:31:53.825 INFO:teuthology.orchestra.run.smithi134.stdout: toolbox x86_64 0.0.99.3-1.module_el8.5.0+877+1c30e0c9 CentOS-AppStream 2.3 M 2021-11-19T08:31:53.825 INFO:teuthology.orchestra.run.smithi134.stdout: udica noarch 0.2.6-2.module_el8.6.0+944+d413f95e CentOS-AppStream 48 k 2021-11-19T08:31:53.825 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2021-11-19T08:31:53.826 INFO:teuthology.orchestra.run.smithi134.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-11-19T08:31:53.826 INFO:teuthology.orchestra.run.smithi134.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-11-19T08:31:53.826 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-11-19T08:31:53.826 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 CentOS-AppStream 25 k 2021-11-19T08:31:53.827 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyxdg noarch 0.25-16.el8 CentOS-AppStream 94 k 2021-11-19T08:31:53.827 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-11-19T08:31:53.827 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-11-19T08:31:53.828 INFO:teuthology.orchestra.run.smithi134.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-11-19T08:31:53.828 INFO:teuthology.orchestra.run.smithi134.stdout:Installing module profiles: 2021-11-19T08:31:53.828 INFO:teuthology.orchestra.run.smithi134.stdout: container-tools/common 2021-11-19T08:31:53.829 INFO:teuthology.orchestra.run.smithi134.stdout:Enabling module streams: 2021-11-19T08:31:53.829 INFO:teuthology.orchestra.run.smithi134.stdout: container-tools rhel8 2021-11-19T08:31:53.829 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:31:53.829 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:31:53.830 INFO:teuthology.orchestra.run.smithi134.stdout:===================================================================================================== 2021-11-19T08:31:53.830 INFO:teuthology.orchestra.run.smithi134.stdout:Install 15 Packages 2021-11-19T08:31:53.830 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 7 Packages 2021-11-19T08:31:53.830 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:31:53.831 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 41 M 2021-11-19T08:31:53.831 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:31:53.881 INFO:teuthology.orchestra.run.smithi134.stdout:(1/22): crun-1.2-1.module_el8.6.0+954+963caf36. 3.4 MB/s | 193 kB 00:00 2021-11-19T08:31:53.888 INFO:teuthology.orchestra.run.smithi134.stdout:(2/22): python3-podman-3.2.1-4.module_el8.6.0+9 20 MB/s | 148 kB 00:00 2021-11-19T08:31:53.891 INFO:teuthology.orchestra.run.smithi134.stdout:(3/22): cockpit-podman-35-1.module_el8.6.0+944+ 6.8 MB/s | 457 kB 00:00 2021-11-19T08:31:53.891 INFO:teuthology.orchestra.run.smithi134.stdout:(4/22): python3-pytoml-0.1.14-5.git7dea353.el8. 9.0 MB/s | 25 kB 00:00 2021-11-19T08:31:53.894 INFO:teuthology.orchestra.run.smithi134.stdout:(5/22): python3-pyxdg-0.25-16.el8.noarch.rpm 27 MB/s | 94 kB 00:00 2021-11-19T08:31:53.983 INFO:teuthology.orchestra.run.smithi134.stdout:(6/22): toolbox-0.0.99.3-1.module_el8.5.0+877+1 25 MB/s | 2.3 MB 00:00 2021-11-19T08:31:53.988 INFO:teuthology.orchestra.run.smithi134.stdout:(7/22): udica-0.2.6-2.module_el8.6.0+944+d413f9 9.8 MB/s | 48 kB 00:00 2021-11-19T08:31:53.992 INFO:teuthology.orchestra.run.smithi134.stdout:(8/22): yajl-2.1.0-10.el8.x86_64.rpm 9.6 MB/s | 41 kB 00:00 2021-11-19T08:31:54.005 INFO:teuthology.orchestra.run.smithi134.stdout:(9/22): python3-chardet-3.0.4-7.el8.noarch.rpm 16 MB/s | 195 kB 00:00 2021-11-19T08:31:54.011 INFO:teuthology.orchestra.run.smithi134.stdout:(10/22): python3-idna-2.5-5.el8.noarch.rpm 16 MB/s | 97 kB 00:00 2021-11-19T08:31:54.015 INFO:teuthology.orchestra.run.smithi134.stdout:(11/22): python3-pysocks-1.6.8-3.el8.noarch.rpm 7.2 MB/s | 34 kB 00:00 2021-11-19T08:31:54.027 INFO:teuthology.orchestra.run.smithi134.stdout:(12/22): python3-requests-2.20.0-2.1.el8_1.noar 11 MB/s | 123 kB 00:00 2021-11-19T08:31:54.050 INFO:teuthology.orchestra.run.smithi134.stdout:(13/22): buildah-1.23.1-2.module_el8.6.0+954+96 35 MB/s | 7.9 MB 00:00 2021-11-19T08:31:54.053 INFO:teuthology.orchestra.run.smithi134.stdout:(14/22): python3-urllib3-1.24.2-5.el8.noarch.rp 6.6 MB/s | 177 kB 00:00 2021-11-19T08:31:54.054 INFO:teuthology.orchestra.run.smithi134.stdout:(15/22): conmon-2.0.30-1.module_el8.6.0+944+d41 15 MB/s | 54 kB 00:00 2021-11-19T08:31:54.068 INFO:teuthology.orchestra.run.smithi134.stdout:(16/22): criu-3.15-3.module_el8.6.0+926+8bef8ae 37 MB/s | 518 kB 00:00 2021-11-19T08:31:54.074 INFO:teuthology.orchestra.run.smithi134.stdout:(17/22): fuse-overlayfs-1.7.1-1.module_el8.6.0+ 13 MB/s | 73 kB 00:00 2021-11-19T08:31:54.081 INFO:teuthology.orchestra.run.smithi134.stdout:(18/22): libslirp-4.4.0-1.module_el8.6.0+926+8b 9.3 MB/s | 70 kB 00:00 2021-11-19T08:31:54.139 INFO:teuthology.orchestra.run.smithi134.stdout:(19/22): skopeo-1.5.0-2.module_el8.6.0+954+963c 27 MB/s | 6.7 MB 00:00 2021-11-19T08:31:54.146 INFO:teuthology.orchestra.run.smithi134.stdout:(20/22): slirp4netns-1.1.8-1.module_el8.6.0+926 7.4 MB/s | 51 kB 00:00 2021-11-19T08:31:54.163 INFO:teuthology.orchestra.run.smithi134.stdout:(21/22): runc-1.0.2-1.module_el8.6.0+926+8bef8a 38 MB/s | 3.1 MB 00:00 2021-11-19T08:31:54.201 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:31:54.204 INFO:teuthology.orchestra.run.smithi023.stdout:===================================================================================================== 2021-11-19T08:31:54.204 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2021-11-19T08:31:54.205 INFO:teuthology.orchestra.run.smithi023.stdout:===================================================================================================== 2021-11-19T08:31:54.205 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrading: 2021-11-19T08:31:54.205 INFO:teuthology.orchestra.run.smithi023.stdout: conmon x86_64 2:2.0.30-1.module_el8.6.0+944+d413f95e CentOS-AppStream 54 k 2021-11-19T08:31:54.205 INFO:teuthology.orchestra.run.smithi023.stdout: containernetworking-plugins x86_64 1.0.1-1.module_el8.6.0+944+d413f95e CentOS-AppStream 19 M 2021-11-19T08:31:54.206 INFO:teuthology.orchestra.run.smithi023.stdout: criu x86_64 3.15-3.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 518 k 2021-11-19T08:31:54.206 INFO:teuthology.orchestra.run.smithi023.stdout: fuse-overlayfs x86_64 1.7.1-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 73 k 2021-11-19T08:31:54.206 INFO:teuthology.orchestra.run.smithi023.stdout: libslirp x86_64 4.4.0-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 70 k 2021-11-19T08:31:54.206 INFO:teuthology.orchestra.run.smithi023.stdout: runc x86_64 1.0.2-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 3.1 M 2021-11-19T08:31:54.206 INFO:teuthology.orchestra.run.smithi023.stdout: slirp4netns x86_64 1.1.8-1.module_el8.6.0+926+8bef8ae7 CentOS-AppStream 51 k 2021-11-19T08:31:54.207 INFO:teuthology.orchestra.run.smithi023.stdout:Installing group/module packages: 2021-11-19T08:31:54.207 INFO:teuthology.orchestra.run.smithi023.stdout: buildah x86_64 1:1.23.1-2.module_el8.6.0+954+963caf36 CentOS-AppStream 7.9 M 2021-11-19T08:31:54.207 INFO:teuthology.orchestra.run.smithi023.stdout: cockpit-podman noarch 35-1.module_el8.6.0+944+d413f95e CentOS-AppStream 457 k 2021-11-19T08:31:54.207 INFO:teuthology.orchestra.run.smithi023.stdout: crun x86_64 1.2-1.module_el8.6.0+954+963caf36 CentOS-AppStream 193 k 2021-11-19T08:31:54.208 INFO:teuthology.orchestra.run.smithi023.stdout: python3-podman noarch 3.2.1-4.module_el8.6.0+954+963caf36 CentOS-AppStream 148 k 2021-11-19T08:31:54.208 INFO:teuthology.orchestra.run.smithi023.stdout: skopeo x86_64 2:1.5.0-2.module_el8.6.0+954+963caf36 CentOS-AppStream 6.7 M 2021-11-19T08:31:54.208 INFO:teuthology.orchestra.run.smithi023.stdout: toolbox x86_64 0.0.99.3-1.module_el8.5.0+877+1c30e0c9 CentOS-AppStream 2.3 M 2021-11-19T08:31:54.208 INFO:teuthology.orchestra.run.smithi023.stdout: udica noarch 0.2.6-2.module_el8.6.0+944+d413f95e CentOS-AppStream 48 k 2021-11-19T08:31:54.209 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2021-11-19T08:31:54.209 INFO:teuthology.orchestra.run.smithi023.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-11-19T08:31:54.209 INFO:teuthology.orchestra.run.smithi023.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-11-19T08:31:54.209 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-11-19T08:31:54.209 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 CentOS-AppStream 25 k 2021-11-19T08:31:54.210 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyxdg noarch 0.25-16.el8 CentOS-AppStream 94 k 2021-11-19T08:31:54.210 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-11-19T08:31:54.210 INFO:teuthology.orchestra.run.smithi023.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-11-19T08:31:54.210 INFO:teuthology.orchestra.run.smithi023.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-11-19T08:31:54.211 INFO:teuthology.orchestra.run.smithi023.stdout:Installing module profiles: 2021-11-19T08:31:54.211 INFO:teuthology.orchestra.run.smithi023.stdout: container-tools/common 2021-11-19T08:31:54.211 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling module streams: 2021-11-19T08:31:54.211 INFO:teuthology.orchestra.run.smithi023.stdout: container-tools rhel8 2021-11-19T08:31:54.212 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:31:54.212 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:31:54.212 INFO:teuthology.orchestra.run.smithi023.stdout:===================================================================================================== 2021-11-19T08:31:54.212 INFO:teuthology.orchestra.run.smithi023.stdout:Install 15 Packages 2021-11-19T08:31:54.212 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrade 7 Packages 2021-11-19T08:31:54.213 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:31:54.213 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 41 M 2021-11-19T08:31:54.214 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:31:54.304 INFO:teuthology.orchestra.run.smithi023.stdout:(1/22): crun-1.2-1.module_el8.6.0+954+963caf36. 2.0 MB/s | 193 kB 00:00 2021-11-19T08:31:54.316 INFO:teuthology.orchestra.run.smithi023.stdout:(2/22): cockpit-podman-35-1.module_el8.6.0+944+ 4.2 MB/s | 457 kB 00:00 2021-11-19T08:31:54.325 INFO:teuthology.orchestra.run.smithi023.stdout:(3/22): python3-podman-3.2.1-4.module_el8.6.0+9 7.0 MB/s | 148 kB 00:00 2021-11-19T08:31:54.325 INFO:teuthology.orchestra.run.smithi023.stdout:(4/22): python3-pytoml-0.1.14-5.git7dea353.el8. 2.7 MB/s | 25 kB 00:00 2021-11-19T08:31:54.338 INFO:teuthology.orchestra.run.smithi023.stdout:(5/22): python3-pyxdg-0.25-16.el8.noarch.rpm 6.9 MB/s | 94 kB 00:00 2021-11-19T08:31:54.447 INFO:teuthology.orchestra.run.smithi023.stdout:(6/22): toolbox-0.0.99.3-1.module_el8.5.0+877+1 21 MB/s | 2.3 MB 00:00 2021-11-19T08:31:54.453 INFO:teuthology.orchestra.run.smithi023.stdout:(7/22): udica-0.2.6-2.module_el8.6.0+944+d413f9 8.2 MB/s | 48 kB 00:00 2021-11-19T08:31:54.460 INFO:teuthology.orchestra.run.smithi023.stdout:(8/22): yajl-2.1.0-10.el8.x86_64.rpm 6.3 MB/s | 41 kB 00:00 2021-11-19T08:31:54.471 INFO:teuthology.orchestra.run.smithi023.stdout:(9/22): python3-chardet-3.0.4-7.el8.noarch.rpm 17 MB/s | 195 kB 00:00 2021-11-19T08:31:54.480 INFO:teuthology.orchestra.run.smithi134.stdout:(22/22): containernetworking-plugins-1.0.1-1.mo 45 MB/s | 19 MB 00:00 2021-11-19T08:31:54.480 INFO:teuthology.orchestra.run.smithi023.stdout:(10/22): python3-idna-2.5-5.el8.noarch.rpm 11 MB/s | 97 kB 00:00 2021-11-19T08:31:54.483 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:31:54.484 INFO:teuthology.orchestra.run.smithi134.stdout:Total 63 MB/s | 41 MB 00:00 2021-11-19T08:31:54.485 INFO:teuthology.orchestra.run.smithi023.stdout:(11/22): python3-pysocks-1.6.8-3.el8.noarch.rpm 7.8 MB/s | 34 kB 00:00 2021-11-19T08:31:54.492 INFO:teuthology.orchestra.run.smithi023.stdout:(12/22): python3-requests-2.20.0-2.1.el8_1.noar 17 MB/s | 123 kB 00:00 2021-11-19T08:31:54.517 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:31:54.517 INFO:teuthology.orchestra.run.smithi023.stdout:(13/22): python3-urllib3-1.24.2-5.el8.noarch.rp 7.0 MB/s | 177 kB 00:00 2021-11-19T08:31:54.529 INFO:teuthology.orchestra.run.smithi023.stdout:(14/22): conmon-2.0.30-1.module_el8.6.0+944+d41 4.4 MB/s | 54 kB 00:00 2021-11-19T08:31:54.544 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:31:54.544 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:31:54.572 INFO:teuthology.orchestra.run.smithi023.stdout:(15/22): buildah-1.23.1-2.module_el8.6.0+954+96 22 MB/s | 7.9 MB 00:00 2021-11-19T08:31:54.594 INFO:teuthology.orchestra.run.smithi023.stdout:(16/22): skopeo-1.5.0-2.module_el8.6.0+954+963c 25 MB/s | 6.7 MB 00:00 2021-11-19T08:31:54.599 INFO:teuthology.orchestra.run.smithi023.stdout:(17/22): criu-3.15-3.module_el8.6.0+926+8bef8ae 19 MB/s | 518 kB 00:00 2021-11-19T08:31:54.657 INFO:teuthology.orchestra.run.smithi023.stdout:(18/22): fuse-overlayfs-1.7.1-1.module_el8.6.0+ 1.1 MB/s | 73 kB 00:00 2021-11-19T08:31:54.658 INFO:teuthology.orchestra.run.smithi023.stdout:(19/22): libslirp-4.4.0-1.module_el8.6.0+926+8b 1.2 MB/s | 70 kB 00:00 2021-11-19T08:31:54.664 INFO:teuthology.orchestra.run.smithi023.stdout:(20/22): slirp4netns-1.1.8-1.module_el8.6.0+926 8.9 MB/s | 51 kB 00:00 2021-11-19T08:31:54.731 INFO:teuthology.orchestra.run.smithi023.stdout:(21/22): runc-1.0.2-1.module_el8.6.0+926+8bef8a 42 MB/s | 3.1 MB 00:00 2021-11-19T08:31:54.886 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:31:54.895 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:31:55.012 INFO:teuthology.orchestra.run.smithi023.stdout:(22/22): containernetworking-plugins-1.0.1-1.mo 40 MB/s | 19 MB 00:00 2021-11-19T08:31:55.015 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:31:55.015 INFO:teuthology.orchestra.run.smithi023.stdout:Total 51 MB/s | 41 MB 00:00 2021-11-19T08:31:55.053 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:31:55.083 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:31:55.084 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:31:55.449 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:31:55.452 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:31:56.195 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:31:56.370 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 1/29 2021-11-19T08:31:56.558 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 2/29 2021-11-19T08:31:56.580 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:31:56.716 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 1/29 2021-11-19T08:31:56.923 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 2/29 2021-11-19T08:31:57.120 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 3/29 2021-11-19T08:31:57.296 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 4/29 2021-11-19T08:31:57.491 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 5/29 2021-11-19T08:31:57.503 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 3/29 2021-11-19T08:31:57.643 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 4/29 2021-11-19T08:31:57.650 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 6/29 2021-11-19T08:31:57.872 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-idna-2.5-5.el8.noarch 7/29 2021-11-19T08:31:57.978 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 5/29 2021-11-19T08:31:58.085 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 8/29 2021-11-19T08:31:58.140 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 6/29 2021-11-19T08:31:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 9/29 2021-11-19T08:31:58.334 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-idna-2.5-5.el8.noarch 7/29 2021-11-19T08:31:58.411 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-19T08:31:58.543 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 8/29 2021-11-19T08:31:58.555 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 11/29 2021-11-19T08:31:58.667 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 9/29 2021-11-19T08:31:58.782 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 12/29 2021-11-19T08:31:58.810 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-19T08:31:58.977 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 11/29 2021-11-19T08:31:58.979 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 13/29 2021-11-19T08:31:59.169 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 12/29 2021-11-19T08:31:59.344 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 13/29 2021-11-19T08:31:59.778 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 14/29 2021-11-19T08:31:59.981 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 15/29 2021-11-19T08:32:00.155 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-19T08:32:00.220 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 14/29 2021-11-19T08:32:00.412 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 15/29 2021-11-19T08:32:00.551 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-19T08:32:01.764 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-19T08:32:01.944 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : containernetworking-plugins-1.0.1-1.module_el8.6.0 17/29 2021-11-19T08:32:02.134 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 18/29 2021-11-19T08:32:02.305 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 16/29 2021-11-19T08:32:02.501 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 19/29 2021-11-19T08:32:02.624 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : containernetworking-plugins-1.0.1-1.module_el8.6.0 17/29 2021-11-19T08:32:02.858 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 18/29 2021-11-19T08:32:03.211 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 19/29 2021-11-19T08:32:03.251 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 20/29 2021-11-19T08:32:03.392 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 21/29 2021-11-19T08:32:03.536 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 22/29 2021-11-19T08:32:03.664 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 23/29 2021-11-19T08:32:03.838 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 24/29 2021-11-19T08:32:04.031 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 25/29 2021-11-19T08:32:04.062 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 20/29 2021-11-19T08:32:04.180 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 26/29 2021-11-19T08:32:04.290 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 21/29 2021-11-19T08:32:04.356 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 27/29 2021-11-19T08:32:04.415 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 22/29 2021-11-19T08:32:04.505 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : containernetworking-plugins-0.9.1-1.module_el8.5.0 28/29 2021-11-19T08:32:04.585 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 23/29 2021-11-19T08:32:04.677 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-19T08:32:04.709 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 24/29 2021-11-19T08:32:04.895 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 25/29 2021-11-19T08:32:05.035 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 26/29 2021-11-19T08:32:05.127 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-19T08:32:05.128 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 1/29 2021-11-19T08:32:05.128 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 2/29 2021-11-19T08:32:05.128 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 3/29 2021-11-19T08:32:05.128 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/29 2021-11-19T08:32:05.129 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 5/29 2021-11-19T08:32:05.129 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 6/29 2021-11-19T08:32:05.129 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 7/29 2021-11-19T08:32:05.129 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 8/29 2021-11-19T08:32:05.129 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 9/29 2021-11-19T08:32:05.130 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-19T08:32:05.130 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 11/29 2021-11-19T08:32:05.130 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-idna-2.5-5.el8.noarch 12/29 2021-11-19T08:32:05.130 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 13/29 2021-11-19T08:32:05.130 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 14/29 2021-11-19T08:32:05.131 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 15/29 2021-11-19T08:32:05.131 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 16/29 2021-11-19T08:32:05.131 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 17/29 2021-11-19T08:32:05.131 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : containernetworking-plugins-1.0.1-1.module_el8.6.0 18/29 2021-11-19T08:32:05.131 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : containernetworking-plugins-0.9.1-1.module_el8.5.0 19/29 2021-11-19T08:32:05.132 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 20/29 2021-11-19T08:32:05.132 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 21/29 2021-11-19T08:32:05.132 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 22/29 2021-11-19T08:32:05.133 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 23/29 2021-11-19T08:32:05.133 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 24/29 2021-11-19T08:32:05.133 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 25/29 2021-11-19T08:32:05.133 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 26/29 2021-11-19T08:32:05.133 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 27/29 2021-11-19T08:32:05.134 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 28/29 2021-11-19T08:32:05.195 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 27/29 2021-11-19T08:32:05.351 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : containernetworking-plugins-0.9.1-1.module_el8.5.0 28/29 2021-11-19T08:32:05.521 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-19T08:32:05.856 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 29/29 2021-11-19T08:32:05.856 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:32:05.857 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2021-11-19T08:32:05.857 INFO:teuthology.orchestra.run.smithi134.stdout: conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-19T08:32:05.857 INFO:teuthology.orchestra.run.smithi134.stdout: containernetworking-plugins-1.0.1-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-19T08:32:05.857 INFO:teuthology.orchestra.run.smithi134.stdout: criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-19T08:32:05.859 INFO:teuthology.orchestra.run.smithi134.stdout: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-19T08:32:05.859 INFO:teuthology.orchestra.run.smithi134.stdout: libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-19T08:32:05.859 INFO:teuthology.orchestra.run.smithi134.stdout: runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-19T08:32:05.859 INFO:teuthology.orchestra.run.smithi134.stdout: slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-19T08:32:05.860 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:32:05.860 INFO:teuthology.orchestra.run.smithi134.stdout: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64 2021-11-19T08:32:05.860 INFO:teuthology.orchestra.run.smithi134.stdout: cockpit-podman-35-1.module_el8.6.0+944+d413f95e.noarch 2021-11-19T08:32:05.860 INFO:teuthology.orchestra.run.smithi134.stdout: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 2021-11-19T08:32:05.861 INFO:teuthology.orchestra.run.smithi134.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-11-19T08:32:05.861 INFO:teuthology.orchestra.run.smithi134.stdout: python3-idna-2.5-5.el8.noarch 2021-11-19T08:32:05.861 INFO:teuthology.orchestra.run.smithi134.stdout: python3-podman-3.2.1-4.module_el8.6.0+954+963caf36.noarch 2021-11-19T08:32:05.861 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-11-19T08:32:05.861 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2021-11-19T08:32:05.862 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyxdg-0.25-16.el8.noarch 2021-11-19T08:32:05.862 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-11-19T08:32:05.862 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-11-19T08:32:05.862 INFO:teuthology.orchestra.run.smithi134.stdout: skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_64 2021-11-19T08:32:05.863 INFO:teuthology.orchestra.run.smithi134.stdout: toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86_64 2021-11-19T08:32:05.863 INFO:teuthology.orchestra.run.smithi134.stdout: udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 2021-11-19T08:32:05.863 INFO:teuthology.orchestra.run.smithi134.stdout: yajl-2.1.0-10.el8.x86_64 2021-11-19T08:32:05.863 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:32:05.863 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:32:06.056 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 29/29 2021-11-19T08:32:06.057 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86 1/29 2021-11-19T08:32:06.057 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cockpit-podman-35-1.module_el8.6.0+944+d413f95e.no 2/29 2021-11-19T08:32:06.057 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 3/29 2021-11-19T08:32:06.057 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-podman-3.2.1-4.module_el8.6.0+954+963caf36 4/29 2021-11-19T08:32:06.058 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 5/29 2021-11-19T08:32:06.058 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 6/29 2021-11-19T08:32:06.058 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_6 7/29 2021-11-19T08:32:06.058 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86 8/29 2021-11-19T08:32:06.059 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 9/29 2021-11-19T08:32:06.059 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/29 2021-11-19T08:32:06.059 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 11/29 2021-11-19T08:32:06.059 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-idna-2.5-5.el8.noarch 12/29 2021-11-19T08:32:06.060 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 13/29 2021-11-19T08:32:06.060 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 14/29 2021-11-19T08:32:06.060 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 15/29 2021-11-19T08:32:06.060 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_ 16/29 2021-11-19T08:32:06.060 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : conmon-2:2.0.27-1.module_el8.5.0+733+9bb5dffa.x86_ 17/29 2021-11-19T08:32:06.061 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : containernetworking-plugins-1.0.1-1.module_el8.6.0 18/29 2021-11-19T08:32:06.061 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : containernetworking-plugins-0.9.1-1.module_el8.5.0 19/29 2021-11-19T08:32:06.061 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 20/29 2021-11-19T08:32:06.061 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : criu-3.15-1.module_el8.5.0+733+9bb5dffa.x86_64 21/29 2021-11-19T08:32:06.062 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7 22/29 2021-11-19T08:32:06.062 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fuse-overlayfs-1.5.0-1.module_el8.5.0+733+9bb5dffa 23/29 2021-11-19T08:32:06.062 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_6 24/29 2021-11-19T08:32:06.062 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libslirp-4.3.1-1.module_el8.5.0+733+9bb5dffa.x86_6 25/29 2021-11-19T08:32:06.063 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 26/29 2021-11-19T08:32:06.063 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : runc-1.0.0-70.rc92.module_el8.5.0+733+9bb5dffa.x86 27/29 2021-11-19T08:32:06.063 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x8 28/29 2021-11-19T08:32:06.165 DEBUG:teuthology.parallel:result is None 2021-11-19T08:32:06.731 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : slirp4netns-1.1.8-1.module_el8.5.0+733+9bb5dffa.x8 29/29 2021-11-19T08:32:06.732 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:32:06.732 INFO:teuthology.orchestra.run.smithi023.stdout:Upgraded: 2021-11-19T08:32:06.732 INFO:teuthology.orchestra.run.smithi023.stdout: conmon-2:2.0.30-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-19T08:32:06.732 INFO:teuthology.orchestra.run.smithi023.stdout: containernetworking-plugins-1.0.1-1.module_el8.6.0+944+d413f95e.x86_64 2021-11-19T08:32:06.733 INFO:teuthology.orchestra.run.smithi023.stdout: criu-3.15-3.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-19T08:32:06.733 INFO:teuthology.orchestra.run.smithi023.stdout: fuse-overlayfs-1.7.1-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-19T08:32:06.733 INFO:teuthology.orchestra.run.smithi023.stdout: libslirp-4.4.0-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-19T08:32:06.733 INFO:teuthology.orchestra.run.smithi023.stdout: runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-19T08:32:06.734 INFO:teuthology.orchestra.run.smithi023.stdout: slirp4netns-1.1.8-1.module_el8.6.0+926+8bef8ae7.x86_64 2021-11-19T08:32:06.734 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:32:06.734 INFO:teuthology.orchestra.run.smithi023.stdout: buildah-1:1.23.1-2.module_el8.6.0+954+963caf36.x86_64 2021-11-19T08:32:06.735 INFO:teuthology.orchestra.run.smithi023.stdout: cockpit-podman-35-1.module_el8.6.0+944+d413f95e.noarch 2021-11-19T08:32:06.735 INFO:teuthology.orchestra.run.smithi023.stdout: crun-1.2-1.module_el8.6.0+954+963caf36.x86_64 2021-11-19T08:32:06.735 INFO:teuthology.orchestra.run.smithi023.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-11-19T08:32:06.735 INFO:teuthology.orchestra.run.smithi023.stdout: python3-idna-2.5-5.el8.noarch 2021-11-19T08:32:06.736 INFO:teuthology.orchestra.run.smithi023.stdout: python3-podman-3.2.1-4.module_el8.6.0+954+963caf36.noarch 2021-11-19T08:32:06.736 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-11-19T08:32:06.736 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2021-11-19T08:32:06.736 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyxdg-0.25-16.el8.noarch 2021-11-19T08:32:06.736 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-11-19T08:32:06.737 INFO:teuthology.orchestra.run.smithi023.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-11-19T08:32:06.737 INFO:teuthology.orchestra.run.smithi023.stdout: skopeo-2:1.5.0-2.module_el8.6.0+954+963caf36.x86_64 2021-11-19T08:32:06.737 INFO:teuthology.orchestra.run.smithi023.stdout: toolbox-0.0.99.3-1.module_el8.5.0+877+1c30e0c9.x86_64 2021-11-19T08:32:06.737 INFO:teuthology.orchestra.run.smithi023.stdout: udica-0.2.6-2.module_el8.6.0+944+d413f95e.noarch 2021-11-19T08:32:06.738 INFO:teuthology.orchestra.run.smithi023.stdout: yajl-2.1.0-10.el8.x86_64 2021-11-19T08:32:06.738 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:32:06.738 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:32:07.033 DEBUG:teuthology.parallel:result is None 2021-11-19T08:32:07.034 INFO:teuthology.run_tasks:Running task install... 2021-11-19T08:32:07.045 DEBUG:teuthology.task.install:project ceph 2021-11-19T08:32:07.045 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e77c7dee6c987c6680b57de9907bbc4d4962f2b1'}} 2021-11-19T08:32:07.046 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'e77c7dee6c987c6680b57de9907bbc4d4962f2b1'} 2021-11-19T08:32:07.046 INFO:teuthology.task.install:Using flavor: default 2021-11-19T08:32:07.053 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-11-19T08:32:07.053 INFO:teuthology.task.install:extra packages: [] 2021-11-19T08:32:07.054 DEBUG:teuthology.orchestra.run.smithi023:> sudo lsb_release -is 2021-11-19T08:32:07.088 INFO:teuthology.orchestra.run.smithi023.stdout:CentOSStream 2021-11-19T08:32:07.090 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-19T08:32:07.090 DEBUG:teuthology.orchestra.run.smithi134:> sudo lsb_release -is 2021-11-19T08:32:07.091 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e77c7dee6c987c6680b57de9907bbc4d4962f2b1', 'tag': 'v16.2.4', 'wait_for_package': False} 2021-11-19T08:32:07.092 DEBUG:teuthology.orchestra.run.smithi023:> sudo lsb_release -is 2021-11-19T08:32:07.127 INFO:teuthology.orchestra.run.smithi134.stdout:CentOSStream 2021-11-19T08:32:07.128 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-19T08:32:07.129 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e77c7dee6c987c6680b57de9907bbc4d4962f2b1', 'tag': 'v16.2.4', 'wait_for_package': False} 2021-11-19T08:32:07.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo lsb_release -is 2021-11-19T08:32:07.163 INFO:teuthology.orchestra.run.smithi023.stdout:CentOSStream 2021-11-19T08:32:07.164 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-19T08:32:07.165 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-19T08:32:07.165 INFO:teuthology.packaging:ref: None 2021-11-19T08:32:07.165 INFO:teuthology.packaging:tag: v16.2.4 2021-11-19T08:32:07.166 INFO:teuthology.packaging:branch: None 2021-11-19T08:32:07.166 INFO:teuthology.packaging:sha1: e77c7dee6c987c6680b57de9907bbc4d4962f2b1 2021-11-19T08:32:07.199 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-19T08:32:07.200 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-19T08:32:07.238 INFO:teuthology.orchestra.run.smithi134.stdout:CentOSStream 2021-11-19T08:32:07.240 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-19T08:32:07.240 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-19T08:32:07.240 INFO:teuthology.packaging:ref: None 2021-11-19T08:32:07.241 INFO:teuthology.packaging:tag: v16.2.4 2021-11-19T08:32:07.241 INFO:teuthology.packaging:branch: None 2021-11-19T08:32:07.241 INFO:teuthology.packaging:sha1: e77c7dee6c987c6680b57de9907bbc4d4962f2b1 2021-11-19T08:32:07.272 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-19T08:32:07.273 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-19T08:32:07.453 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2021-11-19T08:32:07.453 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2021-11-19T08:32:07.486 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2021-11-19T08:32:07.486 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2021-11-19T08:32:07.491 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-11-19T08:32:07.492 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2021-11-19T08:32:07.492 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-11-19T08:32:07.516 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-11-19T08:32:07.516 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2021-11-19T08:32:07.516 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-11-19T08:32:07.524 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-11-19T08:32:07.525 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-19T08:32:07.525 INFO:teuthology.packaging:ref: None 2021-11-19T08:32:07.525 INFO:teuthology.packaging:tag: v16.2.4 2021-11-19T08:32:07.526 INFO:teuthology.packaging:branch: None 2021-11-19T08:32:07.526 INFO:teuthology.packaging:sha1: e77c7dee6c987c6680b57de9907bbc4d4962f2b1 2021-11-19T08:32:07.526 DEBUG:teuthology.orchestra.run.smithi023:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-11-19T08:32:07.542 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-11-19T08:32:07.542 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-19T08:32:07.543 INFO:teuthology.packaging:ref: None 2021-11-19T08:32:07.543 INFO:teuthology.packaging:tag: v16.2.4 2021-11-19T08:32:07.543 INFO:teuthology.packaging:branch: None 2021-11-19T08:32:07.543 INFO:teuthology.packaging:sha1: e77c7dee6c987c6680b57de9907bbc4d4962f2b1 2021-11-19T08:32:07.544 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-11-19T08:32:07.591 DEBUG:teuthology.orchestra.run.smithi023:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-11-19T08:32:07.608 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-11-19T08:32:07.667 DEBUG:teuthology.orchestra.run.smithi023:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-11-19T08:32:07.683 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-11-19T08:32:07.732 INFO:teuthology.orchestra.run.smithi023.stdout:check_obsoletes = 1 2021-11-19T08:32:07.734 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum clean all 2021-11-19T08:32:07.747 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2021-11-19T08:32:07.748 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2021-11-19T08:32:07.980 INFO:teuthology.orchestra.run.smithi023.stdout:75 files removed 2021-11-19T08:32:07.982 INFO:teuthology.orchestra.run.smithi134.stdout:75 files removed 2021-11-19T08:32:08.001 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-radosgw 2021-11-19T08:32:08.002 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2021-11-19T08:32:08.369 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 2.6 MB/s | 74 kB 00:00 2021-11-19T08:32:08.387 INFO:teuthology.orchestra.run.smithi023.stdout:ceph packages for x86_64 2.8 MB/s | 74 kB 00:00 2021-11-19T08:32:08.435 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 517 kB/s | 15 kB 00:00 2021-11-19T08:32:08.455 INFO:teuthology.orchestra.run.smithi023.stdout:ceph noarch packages 503 kB/s | 15 kB 00:00 2021-11-19T08:32:08.493 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 60 kB/s | 1.7 kB 00:00 2021-11-19T08:32:08.517 INFO:teuthology.orchestra.run.smithi023.stdout:ceph source packages 54 kB/s | 1.7 kB 00:00 2021-11-19T08:32:08.931 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-8 - AppStream 41 MB/s | 17 MB 00:00 2021-11-19T08:32:09.002 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-8 - AppStream 37 MB/s | 17 MB 00:00 2021-11-19T08:32:12.155 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-8 - Base 43 MB/s | 15 MB 00:00 2021-11-19T08:32:12.397 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-8 - Base 44 MB/s | 15 MB 00:00 2021-11-19T08:32:14.367 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-8 - Extras 230 kB/s | 16 kB 00:00 2021-11-19T08:32:14.587 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-8 - PowerTools 31 MB/s | 3.6 MB 00:00 2021-11-19T08:32:14.764 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-8 - Extras 238 kB/s | 16 kB 00:00 2021-11-19T08:32:15.002 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-8 - PowerTools 28 MB/s | 3.6 MB 00:00 2021-11-19T08:32:17.564 INFO:teuthology.orchestra.run.smithi023.stdout:Copr repo for python3-asyncssh owned by ceph 1.9 kB/s | 3.5 kB 00:01 2021-11-19T08:32:17.570 INFO:teuthology.orchestra.run.smithi134.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2021-11-19T08:32:18.937 INFO:teuthology.orchestra.run.smithi134.stdout:Apache Arrow 146 kB/s | 184 kB 00:01 2021-11-19T08:32:18.954 INFO:teuthology.orchestra.run.smithi023.stdout:Apache Arrow 144 kB/s | 184 kB 00:01 2021-11-19T08:32:20.191 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 11 MB 00:01 2021-11-19T08:32:20.460 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 8.0 MB/s | 11 MB 00:01 2021-11-19T08:32:22.611 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 3.3 MB/s | 24 kB 00:00 2021-11-19T08:32:22.991 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2021-11-19T08:32:25.399 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:32:25.402 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:32:25.402 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2021-11-19T08:32:25.403 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:32:25.403 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:32:25.404 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2021-11-19T08:32:25.404 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2021-11-19T08:32:25.405 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-19T08:32:25.405 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2021-11-19T08:32:25.405 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2021-11-19T08:32:25.406 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2021-11-19T08:32:25.406 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2021-11-19T08:32:25.407 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2021-11-19T08:32:25.407 INFO:teuthology.orchestra.run.smithi134.stdout: daxctl-libs x86_64 71.1-2.el8 CentOS-Base 41 k 2021-11-19T08:32:25.408 INFO:teuthology.orchestra.run.smithi134.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-11-19T08:32:25.408 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-11-19T08:32:25.408 INFO:teuthology.orchestra.run.smithi134.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-11-19T08:32:25.408 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2021-11-19T08:32:25.409 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-11-19T08:32:25.409 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.6.1-1.el8 CentOS-AppStream 145 k 2021-11-19T08:32:25.409 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-11-19T08:32:25.409 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2021-11-19T08:32:25.410 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2021-11-19T08:32:25.410 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2021-11-19T08:32:25.410 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-11-19T08:32:25.410 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-11-19T08:32:25.411 INFO:teuthology.orchestra.run.smithi134.stdout: ndctl-libs x86_64 71.1-2.el8 CentOS-Base 78 k 2021-11-19T08:32:25.411 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2021-11-19T08:32:25.411 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2021-11-19T08:32:25.411 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2021-11-19T08:32:25.412 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-11-19T08:32:25.412 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2021-11-19T08:32:25.412 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2021-11-19T08:32:25.412 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2021-11-19T08:32:25.412 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:32:25.413 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:32:25.413 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:32:25.413 INFO:teuthology.orchestra.run.smithi134.stdout:Install 25 Packages 2021-11-19T08:32:25.413 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2021-11-19T08:32:25.414 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:32:25.414 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 59 M 2021-11-19T08:32:25.415 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:32:25.601 INFO:teuthology.orchestra.run.smithi134.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 31 MB/s | 6.0 MB 00:00 2021-11-19T08:32:25.604 INFO:teuthology.orchestra.run.smithi134.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 8.2 MB/s | 25 kB 00:00 2021-11-19T08:32:25.630 INFO:teuthology.orchestra.run.smithi134.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 31 MB/s | 801 kB 00:00 2021-11-19T08:32:25.648 INFO:teuthology.orchestra.run.smithi134.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 26 MB/s | 486 kB 00:00 2021-11-19T08:32:25.778 INFO:teuthology.orchestra.run.smithi134.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 32 MB/s | 4.1 MB 00:00 2021-11-19T08:32:25.780 INFO:teuthology.orchestra.run.smithi134.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 16 MB/s | 44 kB 00:00 2021-11-19T08:32:25.783 INFO:teuthology.orchestra.run.smithi134.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 34 MB/s | 80 kB 00:00 2021-11-19T08:32:25.790 INFO:teuthology.orchestra.run.smithi134.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 29 MB/s | 213 kB 00:00 2021-11-19T08:32:25.804 INFO:teuthology.orchestra.run.smithi134.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 27 MB/s | 387 kB 00:00 2021-11-19T08:32:25.817 INFO:teuthology.orchestra.run.smithi134.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 30 MB/s | 368 kB 00:00 2021-11-19T08:32:25.820 INFO:teuthology.orchestra.run.smithi134.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 36 MB/s | 114 kB 00:00 2021-11-19T08:32:25.864 INFO:teuthology.orchestra.run.smithi134.stdout:(12/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 3.3 MB/s | 145 kB 00:00 2021-11-19T08:32:25.873 INFO:teuthology.orchestra.run.smithi134.stdout:(13/27): librdkafka-0.11.4-3.el8.x86_64.rpm 37 MB/s | 354 kB 00:00 2021-11-19T08:32:25.882 INFO:teuthology.orchestra.run.smithi134.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 30 MB/s | 259 kB 00:00 2021-11-19T08:32:25.885 INFO:teuthology.orchestra.run.smithi134.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 15 MB/s | 44 kB 00:00 2021-11-19T08:32:25.972 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:32:25.975 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:32:25.975 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2021-11-19T08:32:25.975 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:32:25.975 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:32:25.976 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2021-11-19T08:32:25.976 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrading: 2021-11-19T08:32:25.976 INFO:teuthology.orchestra.run.smithi023.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-19T08:32:25.976 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2021-11-19T08:32:25.977 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2021-11-19T08:32:25.977 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2021-11-19T08:32:25.977 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2021-11-19T08:32:25.977 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2021-11-19T08:32:25.978 INFO:teuthology.orchestra.run.smithi023.stdout: daxctl-libs x86_64 71.1-2.el8 CentOS-Base 41 k 2021-11-19T08:32:25.978 INFO:teuthology.orchestra.run.smithi023.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-11-19T08:32:25.978 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-11-19T08:32:25.978 INFO:teuthology.orchestra.run.smithi023.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-11-19T08:32:25.979 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2021-11-19T08:32:25.979 INFO:teuthology.orchestra.run.smithi023.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-11-19T08:32:25.979 INFO:teuthology.orchestra.run.smithi023.stdout: libpmemobj x86_64 1.6.1-1.el8 CentOS-AppStream 145 k 2021-11-19T08:32:25.980 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-11-19T08:32:25.980 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2021-11-19T08:32:25.980 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2021-11-19T08:32:25.980 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2021-11-19T08:32:25.981 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-11-19T08:32:25.981 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-11-19T08:32:25.981 INFO:teuthology.orchestra.run.smithi023.stdout: ndctl-libs x86_64 71.1-2.el8 CentOS-Base 78 k 2021-11-19T08:32:25.981 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2021-11-19T08:32:25.981 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2021-11-19T08:32:25.982 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2021-11-19T08:32:25.982 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-11-19T08:32:25.982 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2021-11-19T08:32:25.982 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2021-11-19T08:32:25.983 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2021-11-19T08:32:25.983 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:32:25.983 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:32:25.983 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:32:25.983 INFO:teuthology.orchestra.run.smithi023.stdout:Install 25 Packages 2021-11-19T08:32:25.984 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrade 2 Packages 2021-11-19T08:32:25.984 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:32:25.985 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 59 M 2021-11-19T08:32:25.985 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:32:25.995 INFO:teuthology.orchestra.run.smithi134.stdout:(16/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 22 MB/s | 13 MB 00:00 2021-11-19T08:32:25.995 INFO:teuthology.orchestra.run.smithi134.stdout:(17/27): daxctl-libs-71.1-2.el8.x86_64.rpm 378 kB/s | 41 kB 00:00 2021-11-19T08:32:26.002 INFO:teuthology.orchestra.run.smithi134.stdout:(18/27): ndctl-libs-71.1-2.el8.x86_64.rpm 11 MB/s | 78 kB 00:00 2021-11-19T08:32:26.157 INFO:teuthology.orchestra.run.smithi134.stdout:(19/27): ceph-common-16.2.4-0.el8.x86_64.rpm 32 MB/s | 24 MB 00:00 2021-11-19T08:32:26.226 INFO:teuthology.orchestra.run.smithi023.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 25 MB/s | 6.0 MB 00:00 2021-11-19T08:32:26.229 INFO:teuthology.orchestra.run.smithi023.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 8.1 MB/s | 25 kB 00:00 2021-11-19T08:32:26.229 INFO:teuthology.orchestra.run.smithi134.stdout:(20/27): librabbitmq-0.9.0-3.el8.x86_64.rpm 200 kB/s | 47 kB 00:00 2021-11-19T08:32:26.257 INFO:teuthology.orchestra.run.smithi023.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 801 kB 00:00 2021-11-19T08:32:26.266 INFO:teuthology.orchestra.run.smithi134.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 441 kB/s | 116 kB 00:00 2021-11-19T08:32:26.274 INFO:teuthology.orchestra.run.smithi023.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 29 MB/s | 486 kB 00:00 2021-11-19T08:32:26.296 INFO:teuthology.orchestra.run.smithi134.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 2.0 MB/s | 59 kB 00:00 2021-11-19T08:32:26.324 INFO:teuthology.orchestra.run.smithi134.stdout:(23/27): libunwind-1.3.1-3.el8.x86_64.rpm 2.7 MB/s | 75 kB 00:00 2021-11-19T08:32:26.371 INFO:teuthology.orchestra.run.smithi134.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 1.2 MB/s | 181 kB 00:00 2021-11-19T08:32:26.378 INFO:teuthology.orchestra.run.smithi023.stdout:(5/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 32 MB/s | 13 MB 00:00 2021-11-19T08:32:26.380 INFO:teuthology.orchestra.run.smithi023.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 27 MB/s | 44 kB 00:00 2021-11-19T08:32:26.381 INFO:teuthology.orchestra.run.smithi134.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 65 MB/s | 3.7 MB 00:00 2021-11-19T08:32:26.382 INFO:teuthology.orchestra.run.smithi023.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 34 MB/s | 80 kB 00:00 2021-11-19T08:32:26.383 INFO:teuthology.orchestra.run.smithi134.stdout:(26/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2021-11-19T08:32:26.388 INFO:teuthology.orchestra.run.smithi023.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 42 MB/s | 213 kB 00:00 2021-11-19T08:32:26.398 INFO:teuthology.orchestra.run.smithi023.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 36 MB/s | 387 kB 00:00 2021-11-19T08:32:26.409 INFO:teuthology.orchestra.run.smithi023.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 33 MB/s | 368 kB 00:00 2021-11-19T08:32:26.413 INFO:teuthology.orchestra.run.smithi023.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 38 MB/s | 114 kB 00:00 2021-11-19T08:32:26.459 INFO:teuthology.orchestra.run.smithi023.stdout:(12/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 3.1 MB/s | 145 kB 00:00 2021-11-19T08:32:26.465 INFO:teuthology.orchestra.run.smithi023.stdout:(13/27): librdkafka-0.11.4-3.el8.x86_64.rpm 55 MB/s | 354 kB 00:00 2021-11-19T08:32:26.470 INFO:teuthology.orchestra.run.smithi023.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 55 MB/s | 259 kB 00:00 2021-11-19T08:32:26.471 INFO:teuthology.orchestra.run.smithi023.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 29 MB/s | 44 kB 00:00 2021-11-19T08:32:26.473 INFO:teuthology.orchestra.run.smithi134.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 40 MB/s | 4.0 MB 00:00 2021-11-19T08:32:26.473 INFO:teuthology.orchestra.run.smithi023.stdout:(16/27): daxctl-libs-71.1-2.el8.x86_64.rpm 27 MB/s | 41 kB 00:00 2021-11-19T08:32:26.475 INFO:teuthology.orchestra.run.smithi023.stdout:(17/27): librabbitmq-0.9.0-3.el8.x86_64.rpm 23 MB/s | 47 kB 00:00 2021-11-19T08:32:26.477 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:32:26.478 INFO:teuthology.orchestra.run.smithi134.stdout:Total 55 MB/s | 59 MB 00:01 2021-11-19T08:32:26.478 INFO:teuthology.orchestra.run.smithi023.stdout:(18/27): ndctl-libs-71.1-2.el8.x86_64.rpm 29 MB/s | 78 kB 00:00 2021-11-19T08:32:26.489 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:32:26.497 INFO:teuthology.orchestra.run.smithi023.stdout:(19/27): librgw2-16.2.4-0.el8.x86_64.rpm 18 MB/s | 4.1 MB 00:00 2021-11-19T08:32:26.536 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:32:26.536 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:32:26.619 INFO:teuthology.orchestra.run.smithi023.stdout:(20/27): fmt-6.2.1-1.el8.x86_64.rpm 831 kB/s | 116 kB 00:00 2021-11-19T08:32:26.774 INFO:teuthology.orchestra.run.smithi023.stdout:(21/27): ceph-common-16.2.4-0.el8.x86_64.rpm 30 MB/s | 24 MB 00:00 2021-11-19T08:32:26.838 INFO:teuthology.orchestra.run.smithi023.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 899 kB/s | 306 kB 00:00 2021-11-19T08:32:26.839 INFO:teuthology.orchestra.run.smithi023.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 822 kB/s | 181 kB 00:00 2021-11-19T08:32:26.869 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:32:26.870 INFO:teuthology.orchestra.run.smithi023.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 2.4 MB/s | 75 kB 00:00 2021-11-19T08:32:26.872 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:32:26.888 INFO:teuthology.orchestra.run.smithi023.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 76 MB/s | 3.7 MB 00:00 2021-11-19T08:32:26.933 INFO:teuthology.orchestra.run.smithi023.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 63 MB/s | 4.0 MB 00:00 2021-11-19T08:32:26.934 INFO:teuthology.orchestra.run.smithi023.stdout:(27/27): liboath-2.6.2-3.el8.x86_64.rpm 368 kB/s | 59 kB 00:00 2021-11-19T08:32:26.938 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:32:26.939 INFO:teuthology.orchestra.run.smithi023.stdout:Total 61 MB/s | 59 MB 00:00 2021-11-19T08:32:26.951 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:32:27.003 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:32:27.003 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:32:27.356 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:32:27.359 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:32:28.072 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:32:28.232 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2021-11-19T08:32:28.410 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2021-11-19T08:32:28.584 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-19T08:32:28.668 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:32:28.803 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2021-11-19T08:32:28.960 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2021-11-19T08:32:28.994 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-19T08:32:29.097 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-19T08:32:29.184 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-19T08:32:29.275 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-19T08:32:29.516 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2021-11-19T08:32:29.533 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2021-11-19T08:32:29.675 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-19T08:32:29.705 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-19T08:32:29.754 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-19T08:32:29.810 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2021-11-19T08:32:29.907 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/29 2021-11-19T08:32:30.009 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2021-11-19T08:32:30.058 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-19T08:32:30.130 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-19T08:32:30.195 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2021-11-19T08:32:30.372 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/29 2021-11-19T08:32:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-19T08:32:30.513 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-19T08:32:30.625 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-19T08:32:30.702 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-19T08:32:30.880 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2021-11-19T08:32:30.949 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2021-11-19T08:32:30.983 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-19T08:32:31.050 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-19T08:32:31.063 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-19T08:32:31.127 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2021-11-19T08:32:31.313 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2021-11-19T08:32:31.538 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-19T08:32:31.624 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: daxctl-libs-71.1-2.el8.x86_64 11/29 2021-11-19T08:32:31.713 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2021-11-19T08:32:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2021-11-19T08:32:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2021-11-19T08:32:32.151 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2021-11-19T08:32:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-19T08:32:32.283 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-19T08:32:32.360 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-19T08:32:32.478 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ndctl-libs-71.1-2.el8.x86_64 14/29 2021-11-19T08:32:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-19T08:32:32.638 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-19T08:32:32.932 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-19T08:32:33.074 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-19T08:32:33.081 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2021-11-19T08:32:33.174 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-19T08:32:33.229 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-19T08:32:33.351 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2021-11-19T08:32:33.395 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2021-11-19T08:32:33.517 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2021-11-19T08:32:33.593 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2021-11-19T08:32:33.707 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-19T08:32:33.718 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2021-11-19T08:32:33.775 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-19T08:32:33.804 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-19T08:32:33.859 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2021-11-19T08:32:33.974 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2021-11-19T08:32:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2021-11-19T08:32:34.166 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2021-11-19T08:32:34.205 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2021-11-19T08:32:34.334 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2021-11-19T08:32:34.426 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2021-11-19T08:32:34.533 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2021-11-19T08:32:34.596 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2021-11-19T08:32:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-19T08:32:39.074 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-19T08:32:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-19T08:32:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-19T08:32:39.771 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-19T08:32:40.075 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-19T08:32:40.168 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-19T08:32:40.273 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-19T08:32:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2021-11-19T08:32:40.573 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-19T08:32:40.717 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2021-11-19T08:32:40.836 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-19T08:32:58.260 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-19T08:32:58.261 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2021-11-19T08:32:58.261 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2021-11-19T08:32:58.261 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /dev 2021-11-19T08:32:58.261 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /run 2021-11-19T08:32:58.262 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2021-11-19T08:32:58.262 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2021-11-19T08:32:58.262 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2021-11-19T08:32:58.262 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2021-11-19T08:32:58.263 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /dev 2021-11-19T08:32:58.263 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:32:59.113 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2021-11-19T08:32:59.114 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /sys 2021-11-19T08:32:59.114 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /proc 2021-11-19T08:32:59.114 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /dev 2021-11-19T08:32:59.115 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /run 2021-11-19T08:32:59.115 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /mnt 2021-11-19T08:32:59.116 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /var/tmp 2021-11-19T08:32:59.116 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /home 2021-11-19T08:32:59.116 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /tmp 2021-11-19T08:32:59.117 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /dev 2021-11-19T08:32:59.117 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:32:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-19T08:32:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-19T08:32:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-19T08:32:59.806 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-19T08:32:59.806 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-19T08:33:00.503 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-19T08:33:00.550 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-19T08:33:00.550 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2021-11-19T08:33:00.550 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2021-11-19T08:33:00.551 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2021-11-19T08:33:00.551 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2021-11-19T08:33:00.551 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2021-11-19T08:33:00.551 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2021-11-19T08:33:00.551 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2021-11-19T08:33:00.552 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2021-11-19T08:33:00.552 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2021-11-19T08:33:00.552 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2021-11-19T08:33:00.552 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2021-11-19T08:33:00.553 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2021-11-19T08:33:00.553 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2021-11-19T08:33:00.553 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2021-11-19T08:33:00.553 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2021-11-19T08:33:00.554 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2021-11-19T08:33:00.554 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2021-11-19T08:33:00.554 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : daxctl-libs-71.1-2.el8.x86_64 18/29 2021-11-19T08:33:00.554 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 19/29 2021-11-19T08:33:00.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ndctl-libs-71.1-2.el8.x86_64 20/29 2021-11-19T08:33:00.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2021-11-19T08:33:00.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2021-11-19T08:33:00.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2021-11-19T08:33:00.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2021-11-19T08:33:00.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2021-11-19T08:33:00.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2021-11-19T08:33:00.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2021-11-19T08:33:00.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2021-11-19T08:33:00.573 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2021-11-19T08:33:00.574 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-19T08:33:00.745 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2021-11-19T08:33:00.745 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-19T08:33:01.396 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2021-11-19T08:33:01.397 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:01.397 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2021-11-19T08:33:01.397 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:01.398 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:33:01.398 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:01.398 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:01.398 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:01.399 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:01.399 INFO:teuthology.orchestra.run.smithi134.stdout: daxctl-libs-71.1-2.el8.x86_64 2021-11-19T08:33:01.399 INFO:teuthology.orchestra.run.smithi134.stdout: fmt-6.2.1-1.el8.x86_64 2021-11-19T08:33:01.399 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-11-19T08:33:01.400 INFO:teuthology.orchestra.run.smithi134.stdout: leveldb-1.22-1.el8.x86_64 2021-11-19T08:33:01.400 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:01.400 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.2-3.el8.x86_64 2021-11-19T08:33:01.400 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.6.1-1.el8.x86_64 2021-11-19T08:33:01.401 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-11-19T08:33:01.401 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:01.401 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-0.11.4-3.el8.x86_64 2021-11-19T08:33:01.401 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:01.401 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.3.1-3.el8.x86_64 2021-11-19T08:33:01.402 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-11-19T08:33:01.402 INFO:teuthology.orchestra.run.smithi134.stdout: ndctl-libs-71.1-2.el8.x86_64 2021-11-19T08:33:01.402 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:01.402 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:01.403 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:01.403 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-11-19T08:33:01.403 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:01.403 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:01.404 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:01.404 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:01.404 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:33:01.486 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2021-11-19T08:33:01.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2021-11-19T08:33:01.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2021-11-19T08:33:01.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2021-11-19T08:33:01.487 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2021-11-19T08:33:01.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2021-11-19T08:33:01.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2021-11-19T08:33:01.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2021-11-19T08:33:01.488 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2021-11-19T08:33:01.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2021-11-19T08:33:01.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2021-11-19T08:33:01.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2021-11-19T08:33:01.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2021-11-19T08:33:01.489 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2021-11-19T08:33:01.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2021-11-19T08:33:01.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2021-11-19T08:33:01.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2021-11-19T08:33:01.490 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2021-11-19T08:33:01.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : daxctl-libs-71.1-2.el8.x86_64 18/29 2021-11-19T08:33:01.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 19/29 2021-11-19T08:33:01.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ndctl-libs-71.1-2.el8.x86_64 20/29 2021-11-19T08:33:01.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2021-11-19T08:33:01.491 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2021-11-19T08:33:01.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2021-11-19T08:33:01.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2021-11-19T08:33:01.492 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2021-11-19T08:33:01.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2021-11-19T08:33:01.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2021-11-19T08:33:01.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2021-11-19T08:33:01.636 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2021-11-19T08:33:02.191 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:40 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:33:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2021-11-19T08:33:02.477 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:02.478 INFO:teuthology.orchestra.run.smithi023.stdout:Upgraded: 2021-11-19T08:33:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:02.479 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:33:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:02.479 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: daxctl-libs-71.1-2.el8.x86_64 2021-11-19T08:33:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: fmt-6.2.1-1.el8.x86_64 2021-11-19T08:33:02.480 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-11-19T08:33:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: leveldb-1.22-1.el8.x86_64 2021-11-19T08:33:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: liboath-2.6.2-3.el8.x86_64 2021-11-19T08:33:02.481 INFO:teuthology.orchestra.run.smithi023.stdout: libpmemobj-1.6.1-1.el8.x86_64 2021-11-19T08:33:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-11-19T08:33:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:02.482 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka-0.11.4-3.el8.x86_64 2021-11-19T08:33:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind-1.3.1-3.el8.x86_64 2021-11-19T08:33:02.483 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-11-19T08:33:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: ndctl-libs-71.1-2.el8.x86_64 2021-11-19T08:33:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:02.484 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-11-19T08:33:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:02.487 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:02.488 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:02.488 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:33:02.755 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-test 2021-11-19T08:33:03.332 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:00:40 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:33:03.474 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:33:03.475 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:03.475 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2021-11-19T08:33:03.475 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:03.475 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:33:03.476 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2021-11-19T08:33:03.476 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2021-11-19T08:33:03.476 INFO:teuthology.orchestra.run.smithi134.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2021-11-19T08:33:03.476 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2021-11-19T08:33:03.477 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-11-19T08:33:03.477 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2021-11-19T08:33:03.477 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-11-19T08:33:03.477 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:03.478 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:33:03.478 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:03.478 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2021-11-19T08:33:03.478 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:03.479 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 53 M 2021-11-19T08:33:03.479 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 229 M 2021-11-19T08:33:03.480 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:33:03.505 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 6.0 MB/s | 169 kB 00:00 2021-11-19T08:33:03.533 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): jq-1.6-3.el8.x86_64.rpm 3.5 MB/s | 202 kB 00:00 2021-11-19T08:33:03.544 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 4.8 MB/s | 187 kB 00:00 2021-11-19T08:33:03.547 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 24 MB/s | 323 kB 00:00 2021-11-19T08:33:03.656 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 619 kB/s | 69 kB 00:00 2021-11-19T08:33:04.548 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 48 MB/s | 52 MB 00:01 2021-11-19T08:33:04.552 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:33:04.553 INFO:teuthology.orchestra.run.smithi134.stdout:Total 49 MB/s | 53 MB 00:01 2021-11-19T08:33:04.559 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:33:04.583 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:33:04.584 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:33:04.673 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:33:04.674 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:04.674 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2021-11-19T08:33:04.675 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:04.675 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:33:04.675 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2021-11-19T08:33:04.676 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2021-11-19T08:33:04.676 INFO:teuthology.orchestra.run.smithi023.stdout: jq x86_64 1.6-3.el8 CentOS-AppStream 202 k 2021-11-19T08:33:04.677 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2021-11-19T08:33:04.677 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-11-19T08:33:04.677 INFO:teuthology.orchestra.run.smithi023.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2021-11-19T08:33:04.677 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-11-19T08:33:04.678 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:04.678 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:33:04.678 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:04.678 INFO:teuthology.orchestra.run.smithi023.stdout:Install 6 Packages 2021-11-19T08:33:04.679 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:04.679 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 53 M 2021-11-19T08:33:04.680 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 229 M 2021-11-19T08:33:04.680 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:33:04.705 INFO:teuthology.orchestra.run.smithi023.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 5.9 MB/s | 169 kB 00:00 2021-11-19T08:33:04.753 INFO:teuthology.orchestra.run.smithi023.stdout:(2/6): jq-1.6-3.el8.x86_64.rpm 2.6 MB/s | 202 kB 00:00 2021-11-19T08:33:04.766 INFO:teuthology.orchestra.run.smithi023.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.0 MB/s | 187 kB 00:00 2021-11-19T08:33:04.771 INFO:teuthology.orchestra.run.smithi023.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 17 MB/s | 323 kB 00:00 2021-11-19T08:33:04.868 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:33:04.871 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:33:04.878 INFO:teuthology.orchestra.run.smithi023.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 618 kB/s | 69 kB 00:00 2021-11-19T08:33:05.425 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:33:05.599 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2021-11-19T08:33:05.785 INFO:teuthology.orchestra.run.smithi023.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 47 MB/s | 52 MB 00:01 2021-11-19T08:33:05.789 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:33:05.790 INFO:teuthology.orchestra.run.smithi023.stdout:Total 47 MB/s | 53 MB 00:01 2021-11-19T08:33:05.797 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:33:05.805 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2021-11-19T08:33:05.824 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:33:05.824 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:33:05.994 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-19T08:33:06.076 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-19T08:33:06.124 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:33:06.128 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:33:06.268 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2021-11-19T08:33:06.419 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-19T08:33:06.657 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:33:06.874 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 1/6 2021-11-19T08:33:07.029 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2021-11-19T08:33:07.199 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-19T08:33:07.279 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2021-11-19T08:33:07.433 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : jq-1.6-3.el8.x86_64 4/6 2021-11-19T08:33:07.592 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-19T08:33:11.233 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-19T08:33:11.492 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-19T08:33:11.751 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-19T08:33:11.752 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2021-11-19T08:33:11.752 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2021-11-19T08:33:11.752 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2021-11-19T08:33:11.753 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-11-19T08:33:11.753 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2021-11-19T08:33:12.182 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-11-19T08:33:12.182 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:12.182 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:33:12.183 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-3.el8.x86_64 2021-11-19T08:33:12.183 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-11-19T08:33:12.183 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-11-19T08:33:12.183 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:12.184 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:33:12.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2021-11-19T08:33:12.698 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2021-11-19T08:33:12.905 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:50 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:33:12.914 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-19T08:33:13.176 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2021-11-19T08:33:13.176 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2021-11-19T08:33:13.176 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2021-11-19T08:33:13.177 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : jq-1.6-3.el8.x86_64 3/6 2021-11-19T08:33:13.177 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-11-19T08:33:13.177 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2021-11-19T08:33:13.609 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-11-19T08:33:13.610 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:13.610 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:33:13.610 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-3.el8.x86_64 2021-11-19T08:33:13.610 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-11-19T08:33:13.611 INFO:teuthology.orchestra.run.smithi023.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-11-19T08:33:13.611 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:13.611 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:33:13.804 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph 2021-11-19T08:33:14.225 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:33:14.228 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:14.229 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2021-11-19T08:33:14.229 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:14.229 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:33:14.229 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2021-11-19T08:33:14.230 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2021-11-19T08:33:14.230 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2021-11-19T08:33:14.231 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2021-11-19T08:33:14.231 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2021-11-19T08:33:14.232 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2021-11-19T08:33:14.232 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2021-11-19T08:33:14.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-11-19T08:33:14.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-11-19T08:33:14.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-11-19T08:33:14.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-11-19T08:33:14.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-11-19T08:33:14.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2021-11-19T08:33:14.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-11-19T08:33:14.234 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-11-19T08:33:14.234 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-11-19T08:33:14.235 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-11-19T08:33:14.235 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-11-19T08:33:14.235 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-11-19T08:33:14.235 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-11-19T08:33:14.235 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-11-19T08:33:14.236 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-11-19T08:33:14.236 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-11-19T08:33:14.236 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-11-19T08:33:14.236 INFO:teuthology.orchestra.run.smithi134.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-11-19T08:33:14.237 INFO:teuthology.orchestra.run.smithi134.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-11-19T08:33:14.237 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-11-19T08:33:14.237 INFO:teuthology.orchestra.run.smithi134.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-11-19T08:33:14.237 INFO:teuthology.orchestra.run.smithi134.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-11-19T08:33:14.237 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-11-19T08:33:14.238 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-11-19T08:33:14.238 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-11-19T08:33:14.238 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-11-19T08:33:14.238 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:14.239 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:33:14.239 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:14.239 INFO:teuthology.orchestra.run.smithi134.stdout:Install 32 Packages 2021-11-19T08:33:14.239 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:14.240 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 31 M 2021-11-19T08:33:14.240 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 111 M 2021-11-19T08:33:14.240 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:33:14.254 INFO:teuthology.orchestra.run.smithi134.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 321 kB/s | 6.3 kB 00:00 2021-11-19T08:33:14.304 INFO:teuthology.orchestra.run.smithi134.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 22 MB/s | 1.6 MB 00:00 2021-11-19T08:33:14.312 INFO:teuthology.orchestra.run.smithi134.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 31 MB/s | 2.4 MB 00:00 2021-11-19T08:33:14.319 INFO:teuthology.orchestra.run.smithi134.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 30 MB/s | 220 kB 00:00 2021-11-19T08:33:14.351 INFO:teuthology.orchestra.run.smithi134.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 4.9 MB/s | 157 kB 00:00 2021-11-19T08:33:14.352 INFO:teuthology.orchestra.run.smithi134.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 29 MB/s | 39 kB 00:00 2021-11-19T08:33:14.355 INFO:teuthology.orchestra.run.smithi134.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 47 MB/s | 103 kB 00:00 2021-11-19T08:33:14.356 INFO:teuthology.orchestra.run.smithi134.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 35 MB/s | 54 kB 00:00 2021-11-19T08:33:14.364 INFO:teuthology.orchestra.run.smithi134.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 65 MB/s | 457 kB 00:00 2021-11-19T08:33:14.371 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:00:51 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:33:14.372 INFO:teuthology.orchestra.run.smithi134.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 57 MB/s | 237 kB 00:00 2021-11-19T08:33:14.383 INFO:teuthology.orchestra.run.smithi134.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 40 MB/s | 559 kB 00:00 2021-11-19T08:33:14.384 INFO:teuthology.orchestra.run.smithi134.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 49 MB/s | 109 kB 00:00 2021-11-19T08:33:14.464 INFO:teuthology.orchestra.run.smithi134.stdout:(13/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 24 MB/s | 5.0 MB 00:00 2021-11-19T08:33:14.496 INFO:teuthology.orchestra.run.smithi134.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 392 kB/s | 44 kB 00:00 2021-11-19T08:33:14.682 INFO:teuthology.orchestra.run.smithi134.stdout:(15/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 48 MB/s | 18 MB 00:00 2021-11-19T08:33:14.683 INFO:teuthology.orchestra.run.smithi134.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 927 kB/s | 173 kB 00:00 2021-11-19T08:33:14.683 INFO:teuthology.orchestra.run.smithi134.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 843 kB/s | 185 kB 00:00 2021-11-19T08:33:14.713 INFO:teuthology.orchestra.run.smithi134.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 597 kB/s | 18 kB 00:00 2021-11-19T08:33:14.741 INFO:teuthology.orchestra.run.smithi134.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 6.4 MB/s | 384 kB 00:00 2021-11-19T08:33:14.741 INFO:teuthology.orchestra.run.smithi134.stdout:(20/32): python3-jaraco-6.2-6.el8.noarch.rpm 192 kB/s | 11 kB 00:00 2021-11-19T08:33:14.742 INFO:teuthology.orchestra.run.smithi134.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 1.7 MB/s | 49 kB 00:00 2021-11-19T08:33:14.769 INFO:teuthology.orchestra.run.smithi134.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2021-11-19T08:33:14.774 INFO:teuthology.orchestra.run.smithi134.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 559 kB/s | 16 kB 00:00 2021-11-19T08:33:14.797 INFO:teuthology.orchestra.run.smithi134.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 675 kB/s | 19 kB 00:00 2021-11-19T08:33:14.798 INFO:teuthology.orchestra.run.smithi134.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 860 kB/s | 24 kB 00:00 2021-11-19T08:33:14.826 INFO:teuthology.orchestra.run.smithi134.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2021-11-19T08:33:14.827 INFO:teuthology.orchestra.run.smithi134.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 972 kB/s | 27 kB 00:00 2021-11-19T08:33:14.852 INFO:teuthology.orchestra.run.smithi134.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2021-11-19T08:33:14.857 INFO:teuthology.orchestra.run.smithi134.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 7.4 MB/s | 241 kB 00:00 2021-11-19T08:33:14.881 INFO:teuthology.orchestra.run.smithi134.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.9 MB/s | 85 kB 00:00 2021-11-19T08:33:14.886 INFO:teuthology.orchestra.run.smithi134.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.1 MB/s | 251 kB 00:00 2021-11-19T08:33:14.892 INFO:teuthology.orchestra.run.smithi134.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 762 kB/s | 23 kB 00:00 2021-11-19T08:33:14.892 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:33:14.893 INFO:teuthology.orchestra.run.smithi134.stdout:Total 47 MB/s | 31 MB 00:00 2021-11-19T08:33:14.905 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:33:14.944 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:33:14.944 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:33:15.162 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:33:15.165 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:33:15.743 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:33:15.747 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:15.747 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2021-11-19T08:33:15.748 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:15.748 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:33:15.748 INFO:teuthology.orchestra.run.smithi023.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2021-11-19T08:33:15.749 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2021-11-19T08:33:15.749 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2021-11-19T08:33:15.749 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2021-11-19T08:33:15.749 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2021-11-19T08:33:15.750 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2021-11-19T08:33:15.750 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2021-11-19T08:33:15.750 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-11-19T08:33:15.750 INFO:teuthology.orchestra.run.smithi023.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-11-19T08:33:15.750 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-11-19T08:33:15.751 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-11-19T08:33:15.751 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-11-19T08:33:15.751 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-Base 559 k 2021-11-19T08:33:15.752 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-11-19T08:33:15.752 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-11-19T08:33:15.752 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-11-19T08:33:15.752 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-11-19T08:33:15.753 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-11-19T08:33:15.753 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-11-19T08:33:15.753 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-11-19T08:33:15.753 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-11-19T08:33:15.754 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-11-19T08:33:15.754 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-11-19T08:33:15.754 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-11-19T08:33:15.754 INFO:teuthology.orchestra.run.smithi023.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-11-19T08:33:15.755 INFO:teuthology.orchestra.run.smithi023.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-11-19T08:33:15.755 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-11-19T08:33:15.755 INFO:teuthology.orchestra.run.smithi023.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-11-19T08:33:15.755 INFO:teuthology.orchestra.run.smithi023.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-11-19T08:33:15.756 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-11-19T08:33:15.756 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-11-19T08:33:15.756 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-11-19T08:33:15.757 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-11-19T08:33:15.757 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:15.758 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:33:15.758 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:15.759 INFO:teuthology.orchestra.run.smithi023.stdout:Install 32 Packages 2021-11-19T08:33:15.759 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:15.760 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 31 M 2021-11-19T08:33:15.761 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 111 M 2021-11-19T08:33:15.761 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:33:15.773 INFO:teuthology.orchestra.run.smithi023.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 313 kB/s | 6.3 kB 00:00 2021-11-19T08:33:15.831 INFO:teuthology.orchestra.run.smithi023.stdout:(2/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 31 MB/s | 2.4 MB 00:00 2021-11-19T08:33:15.837 INFO:teuthology.orchestra.run.smithi023.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2021-11-19T08:33:15.845 INFO:teuthology.orchestra.run.smithi023.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 27 MB/s | 220 kB 00:00 2021-11-19T08:33:15.882 INFO:teuthology.orchestra.run.smithi023.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 4.1 MB/s | 157 kB 00:00 2021-11-19T08:33:15.884 INFO:teuthology.orchestra.run.smithi023.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 31 MB/s | 39 kB 00:00 2021-11-19T08:33:15.886 INFO:teuthology.orchestra.run.smithi023.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 52 MB/s | 103 kB 00:00 2021-11-19T08:33:15.888 INFO:teuthology.orchestra.run.smithi023.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 32 MB/s | 54 kB 00:00 2021-11-19T08:33:15.901 INFO:teuthology.orchestra.run.smithi023.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 37 MB/s | 457 kB 00:00 2021-11-19T08:33:15.907 INFO:teuthology.orchestra.run.smithi023.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 41 MB/s | 237 kB 00:00 2021-11-19T08:33:15.920 INFO:teuthology.orchestra.run.smithi023.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 41 MB/s | 559 kB 00:00 2021-11-19T08:33:15.922 INFO:teuthology.orchestra.run.smithi023.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 44 MB/s | 109 kB 00:00 2021-11-19T08:33:15.971 INFO:teuthology.orchestra.run.smithi023.stdout:(13/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 25 MB/s | 5.0 MB 00:00 2021-11-19T08:33:16.037 INFO:teuthology.orchestra.run.smithi023.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 383 kB/s | 44 kB 00:00 2021-11-19T08:33:16.202 INFO:teuthology.orchestra.run.smithi023.stdout:(15/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 49 MB/s | 18 MB 00:00 2021-11-19T08:33:16.203 INFO:teuthology.orchestra.run.smithi023.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 796 kB/s | 185 kB 00:00 2021-11-19T08:33:16.204 INFO:teuthology.orchestra.run.smithi023.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.0 MB/s | 173 kB 00:00 2021-11-19T08:33:16.231 INFO:teuthology.orchestra.run.smithi023.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 399 kB/s | 11 kB 00:00 2021-11-19T08:33:16.232 INFO:teuthology.orchestra.run.smithi023.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 630 kB/s | 18 kB 00:00 2021-11-19T08:33:16.260 INFO:teuthology.orchestra.run.smithi023.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.7 MB/s | 49 kB 00:00 2021-11-19T08:33:16.261 INFO:teuthology.orchestra.run.smithi023.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2021-11-19T08:33:16.289 INFO:teuthology.orchestra.run.smithi023.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 550 kB/s | 16 kB 00:00 2021-11-19T08:33:16.318 INFO:teuthology.orchestra.run.smithi023.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 4.9 MB/s | 283 kB 00:00 2021-11-19T08:33:16.318 INFO:teuthology.orchestra.run.smithi023.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 668 kB/s | 19 kB 00:00 2021-11-19T08:33:16.331 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:33:16.345 INFO:teuthology.orchestra.run.smithi023.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 855 kB/s | 24 kB 00:00 2021-11-19T08:33:16.347 INFO:teuthology.orchestra.run.smithi023.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2021-11-19T08:33:16.374 INFO:teuthology.orchestra.run.smithi023.stdout:(27/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.2 MB/s | 384 kB 00:00 2021-11-19T08:33:16.375 INFO:teuthology.orchestra.run.smithi023.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 948 kB/s | 27 kB 00:00 2021-11-19T08:33:16.402 INFO:teuthology.orchestra.run.smithi023.stdout:(29/32): python3-waitress-1.2.1-2.el8.1.noarch. 4.2 MB/s | 241 kB 00:00 2021-11-19T08:33:16.404 INFO:teuthology.orchestra.run.smithi023.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.9 MB/s | 85 kB 00:00 2021-11-19T08:33:16.406 INFO:teuthology.orchestra.run.smithi023.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 7.6 MB/s | 251 kB 00:00 2021-11-19T08:33:16.431 INFO:teuthology.orchestra.run.smithi023.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 789 kB/s | 23 kB 00:00 2021-11-19T08:33:16.434 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:33:16.435 INFO:teuthology.orchestra.run.smithi023.stdout:Total 45 MB/s | 31 MB 00:00 2021-11-19T08:33:16.450 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:33:16.494 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:33:16.495 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:33:16.559 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-11-19T08:33:16.728 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:33:16.731 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:33:16.776 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-11-19T08:33:17.009 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2021-11-19T08:33:17.233 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2021-11-19T08:33:17.405 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2021-11-19T08:33:17.561 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2021-11-19T08:33:17.761 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2021-11-19T08:33:17.890 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:33:17.973 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2021-11-19T08:33:18.086 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-11-19T08:33:18.157 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2021-11-19T08:33:18.299 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-11-19T08:33:18.412 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2021-11-19T08:33:18.566 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2021-11-19T08:33:18.651 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2021-11-19T08:33:18.747 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2021-11-19T08:33:18.885 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2021-11-19T08:33:18.911 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2021-11-19T08:33:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2021-11-19T08:33:19.318 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2021-11-19T08:33:19.321 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2021-11-19T08:33:19.506 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2021-11-19T08:33:19.571 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2021-11-19T08:33:19.705 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2021-11-19T08:33:19.775 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2021-11-19T08:33:19.959 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2021-11-19T08:33:20.028 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2021-11-19T08:33:20.221 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2021-11-19T08:33:20.243 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-19T08:33:20.454 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2021-11-19T08:33:20.582 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2021-11-19T08:33:20.815 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2021-11-19T08:33:20.864 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2021-11-19T08:33:20.986 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2021-11-19T08:33:21.137 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2021-11-19T08:33:21.207 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-11-19T08:33:21.323 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2021-11-19T08:33:21.491 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-11-19T08:33:21.588 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2021-11-19T08:33:21.781 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-11-19T08:33:21.791 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-19T08:33:22.074 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2021-11-19T08:33:22.084 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2021-11-19T08:33:22.313 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2021-11-19T08:33:22.426 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2021-11-19T08:33:22.484 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2021-11-19T08:33:22.645 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-11-19T08:33:22.778 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-11-19T08:33:22.925 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-11-19T08:33:23.106 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2021-11-19T08:33:23.178 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-11-19T08:33:23.343 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-19T08:33:23.399 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2021-11-19T08:33:23.768 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2021-11-19T08:33:24.098 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-11-19T08:33:24.447 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2021-11-19T08:33:24.666 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-19T08:33:24.864 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-19T08:33:25.118 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-19T08:33:25.552 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-19T08:33:25.801 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-19T08:33:26.024 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-19T08:33:26.266 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-19T08:33:26.283 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-19T08:33:26.290 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2021-11-19T08:33:26.457 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-19T08:33:26.524 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-19T08:33:26.856 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-19T08:33:26.856 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2021-11-19T08:33:26.856 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2021-11-19T08:33:26.857 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2021-11-19T08:33:26.857 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2021-11-19T08:33:26.857 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2021-11-19T08:33:26.857 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2021-11-19T08:33:26.858 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2021-11-19T08:33:26.858 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2021-11-19T08:33:26.858 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2021-11-19T08:33:26.858 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2021-11-19T08:33:26.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-11-19T08:33:26.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2021-11-19T08:33:26.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2021-11-19T08:33:26.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2021-11-19T08:33:26.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2021-11-19T08:33:26.860 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2021-11-19T08:33:26.860 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-19T08:33:26.860 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2021-11-19T08:33:26.860 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2021-11-19T08:33:26.861 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2021-11-19T08:33:26.861 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2021-11-19T08:33:26.861 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2021-11-19T08:33:26.861 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2021-11-19T08:33:26.862 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2021-11-19T08:33:26.862 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2021-11-19T08:33:26.862 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2021-11-19T08:33:26.862 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2021-11-19T08:33:26.863 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2021-11-19T08:33:26.863 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2021-11-19T08:33:26.864 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2021-11-19T08:33:26.864 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2021-11-19T08:33:26.985 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2021-11-19T08:33:27.216 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-19T08:33:27.452 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2021-11-19T08:33:27.639 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-19T08:33:27.657 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2021-11-19T08:33:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2021-11-19T08:33:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:27.668 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:33:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2021-11-19T08:33:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-11-19T08:33:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-11-19T08:33:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-11-19T08:33:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-11-19T08:33:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-11-19T08:33:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2021-11-19T08:33:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-6.2-6.el8.noarch 2021-11-19T08:33:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-11-19T08:33:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-11-19T08:33:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.0.6-13.el8.noarch 2021-11-19T08:33:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-11-19T08:33:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-11-19T08:33:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-11-19T08:33:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-2.6-1.el8.noarch 2021-11-19T08:33:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-11-19T08:33:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.14-14.el8.noarch 2021-11-19T08:33:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2017.2-9.el8.noarch 2021-11-19T08:33:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-11-19T08:33:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-11-19T08:33:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-11-19T08:33:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-11-19T08:33:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-11-19T08:33:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-11-19T08:33:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-11-19T08:33:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-11-19T08:33:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-11-19T08:33:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:27.678 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:33:27.813 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-19T08:33:27.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2021-11-19T08:33:28.221 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2021-11-19T08:33:28.221 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2021-11-19T08:33:28.221 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2021-11-19T08:33:28.221 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2021-11-19T08:33:28.222 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2021-11-19T08:33:28.222 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2021-11-19T08:33:28.222 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2021-11-19T08:33:28.223 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2021-11-19T08:33:28.223 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2021-11-19T08:33:28.223 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2021-11-19T08:33:28.223 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2021-11-19T08:33:28.224 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-11-19T08:33:28.224 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2021-11-19T08:33:28.224 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2021-11-19T08:33:28.225 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2021-11-19T08:33:28.225 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2021-11-19T08:33:28.225 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2021-11-19T08:33:28.226 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2021-11-19T08:33:28.226 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2021-11-19T08:33:28.226 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2021-11-19T08:33:28.227 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2021-11-19T08:33:28.227 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2021-11-19T08:33:28.228 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2021-11-19T08:33:28.228 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2021-11-19T08:33:28.228 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2021-11-19T08:33:28.228 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2021-11-19T08:33:28.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2021-11-19T08:33:28.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2021-11-19T08:33:28.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2021-11-19T08:33:28.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2021-11-19T08:33:28.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2021-11-19T08:33:28.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2021-11-19T08:33:28.487 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:06 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:33:28.983 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2021-11-19T08:33:28.984 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:28.984 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:33:28.984 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:28.984 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:28.984 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:28.985 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2021-11-19T08:33:28.985 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:28.986 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:28.986 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-11-19T08:33:28.986 INFO:teuthology.orchestra.run.smithi023.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-11-19T08:33:28.986 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-11-19T08:33:28.987 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-11-19T08:33:28.987 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-11-19T08:33:28.988 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2021-11-19T08:33:28.988 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-6.2-6.el8.noarch 2021-11-19T08:33:28.989 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-11-19T08:33:28.989 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-11-19T08:33:28.989 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako-1.0.6-13.el8.noarch 2021-11-19T08:33:28.989 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-11-19T08:33:28.990 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-11-19T08:33:28.990 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-11-19T08:33:28.990 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend-2.6-1.el8.noarch 2021-11-19T08:33:28.990 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-11-19T08:33:28.991 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser-2.14-14.el8.noarch 2021-11-19T08:33:28.991 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz-2017.2-9.el8.noarch 2021-11-19T08:33:28.991 INFO:teuthology.orchestra.run.smithi023.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-11-19T08:33:28.991 INFO:teuthology.orchestra.run.smithi023.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-11-19T08:33:28.992 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-11-19T08:33:28.992 INFO:teuthology.orchestra.run.smithi023.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-11-19T08:33:28.992 INFO:teuthology.orchestra.run.smithi023.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-11-19T08:33:28.992 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-11-19T08:33:28.993 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-11-19T08:33:28.993 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-11-19T08:33:28.993 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-11-19T08:33:28.993 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:28.994 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:33:29.217 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-base 2021-11-19T08:33:29.504 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:33:29.787 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:06 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:33:29.797 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:33:29.798 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2021-11-19T08:33:29.799 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:33:29.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2021-11-19T08:33:30.438 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:08 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:33:30.857 INFO:teuthology.orchestra.run.smithi023.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:33:31.164 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:33:31.165 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2021-11-19T08:33:31.165 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:33:31.244 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install cephadm 2021-11-19T08:33:31.763 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:33:31.764 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:31.764 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2021-11-19T08:33:31.764 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:31.765 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:33:31.765 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2021-11-19T08:33:31.765 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:31.765 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:33:31.766 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:31.766 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2021-11-19T08:33:31.766 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:31.766 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 71 k 2021-11-19T08:33:31.767 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 290 k 2021-11-19T08:33:31.767 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:33:31.777 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-16.2.4-0.el8.noarch.rpm 5.7 MB/s | 71 kB 00:00 2021-11-19T08:33:31.778 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:33:31.778 INFO:teuthology.orchestra.run.smithi134.stdout:Total 5.1 MB/s | 71 kB 00:00 2021-11-19T08:33:31.778 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:33:31.786 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:33:31.786 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:33:31.787 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:33:31.791 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:33:31.808 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:08 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:33:31.861 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:33:33.173 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:33:33.174 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:33.175 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2021-11-19T08:33:33.175 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:33.176 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:33:33.176 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2021-11-19T08:33:33.176 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:33.177 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:33:33.178 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:33.178 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2021-11-19T08:33:33.178 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:33.179 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 71 k 2021-11-19T08:33:33.179 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 290 k 2021-11-19T08:33:33.179 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:33:33.188 INFO:teuthology.orchestra.run.smithi023.stdout:cephadm-16.2.4-0.el8.noarch.rpm 5.4 MB/s | 71 kB 00:00 2021-11-19T08:33:33.188 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:33:33.189 INFO:teuthology.orchestra.run.smithi023.stdout:Total 4.9 MB/s | 71 kB 00:00 2021-11-19T08:33:33.190 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:33:33.197 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:33:33.197 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:33:33.198 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:33:33.202 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:33:33.248 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:33:34.541 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-19T08:33:34.846 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-19T08:33:35.043 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-19T08:33:35.409 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-19T08:33:35.410 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:35.410 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:33:35.410 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:16.2.4-0.el8.noarch 2021-11-19T08:33:35.410 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:35.411 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:33:35.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2021-11-19T08:33:36.173 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:14 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:33:36.314 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-19T08:33:36.578 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-19T08:33:36.782 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-19T08:33:37.133 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2021-11-19T08:33:37.133 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:37.133 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:33:37.134 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm-2:16.2.4-0.el8.noarch 2021-11-19T08:33:37.134 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:37.134 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:33:37.335 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-immutable-object-cache 2021-11-19T08:33:37.457 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:33:37.458 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:37.458 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2021-11-19T08:33:37.458 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:37.458 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:33:37.459 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2021-11-19T08:33:37.459 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:37.459 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:33:37.459 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:37.460 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2021-11-19T08:33:37.460 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:37.460 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 163 k 2021-11-19T08:33:37.460 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 587 k 2021-11-19T08:33:37.460 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:33:37.476 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 8.8 MB/s | 163 kB 00:00 2021-11-19T08:33:37.477 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:33:37.478 INFO:teuthology.orchestra.run.smithi134.stdout:Total 8.1 MB/s | 163 kB 00:00 2021-11-19T08:33:37.478 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:33:37.491 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:33:37.491 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:33:37.539 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:33:37.542 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:33:37.651 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:33:37.881 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:33:37.917 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:14 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:33:38.295 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:33:38.608 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:33:38.608 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:38.609 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:33:38.609 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:38.609 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:38.609 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:33:39.282 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repo Size 2021-11-19T08:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2021-11-19T08:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2021-11-19T08:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 163 k 2021-11-19T08:33:39.286 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 587 k 2021-11-19T08:33:39.286 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 12 MB/s | 163 kB 00:00 2021-11-19T08:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout:Total 11 MB/s | 163 kB 00:00 2021-11-19T08:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:33:39.680 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2021-11-19T08:33:40.283 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:18 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:33:40.844 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:33:41.403 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:33:41.713 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:33:41.713 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:41.713 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:33:41.713 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2021-11-19T08:33:41.714 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:41.714 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:33:41.744 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:33:41.914 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-mgr 2021-11-19T08:33:42.039 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:33:42.040 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2021-11-19T08:33:42.040 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:33:42.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2021-11-19T08:33:42.489 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:19 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:33:42.675 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:20 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:33:43.579 INFO:teuthology.orchestra.run.smithi023.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:33:43.884 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:33:43.886 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2021-11-19T08:33:43.886 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:33:43.955 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:33:43.956 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:43.956 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2021-11-19T08:33:43.956 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:43.957 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:33:43.957 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2021-11-19T08:33:43.957 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2021-11-19T08:33:43.957 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2021-11-19T08:33:43.957 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2021-11-19T08:33:43.958 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-11-19T08:33:43.958 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-11-19T08:33:43.958 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-11-19T08:33:43.958 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:43.959 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:33:43.959 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:33:43.959 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2021-11-19T08:33:43.959 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:43.960 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.0 M 2021-11-19T08:33:43.960 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 15 M 2021-11-19T08:33:43.961 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:33:43.969 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-mgr-dashboard 2021-11-19T08:33:43.979 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 1.1 MB/s | 24 kB 00:00 2021-11-19T08:33:43.987 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 312 kB/s | 8.9 kB 00:00 2021-11-19T08:33:44.012 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 49 MB/s | 2.7 MB 00:00 2021-11-19T08:33:44.016 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.1 MB/s | 43 kB 00:00 2021-11-19T08:33:44.428 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 76 kB/s | 34 kB 00:00 2021-11-19T08:33:44.483 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 418 kB/s | 196 kB 00:00 2021-11-19T08:33:44.486 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:33:44.486 INFO:teuthology.orchestra.run.smithi134.stdout:Total 5.6 MB/s | 3.0 MB 00:00 2021-11-19T08:33:44.489 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:33:44.499 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:33:44.499 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:33:44.535 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:21 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:33:44.572 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:33:44.575 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:33:44.878 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:33:45.163 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-11-19T08:33:45.399 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-11-19T08:33:45.599 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-11-19T08:33:45.702 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2021-11-19T08:33:46.000 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:33:46.001 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:46.001 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2021-11-19T08:33:46.001 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:46.002 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:33:46.002 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2021-11-19T08:33:46.006 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2021-11-19T08:33:46.007 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2021-11-19T08:33:46.007 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2021-11-19T08:33:46.007 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-11-19T08:33:46.008 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-11-19T08:33:46.008 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-11-19T08:33:46.008 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:46.008 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:33:46.008 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:33:46.009 INFO:teuthology.orchestra.run.smithi023.stdout:Install 6 Packages 2021-11-19T08:33:46.009 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:46.010 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 3.0 M 2021-11-19T08:33:46.010 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 15 M 2021-11-19T08:33:46.010 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:33:46.025 INFO:teuthology.orchestra.run.smithi023.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 1.1 MB/s | 24 kB 00:00 2021-11-19T08:33:46.026 INFO:teuthology.orchestra.run.smithi023.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 391 kB/s | 8.9 kB 00:00 2021-11-19T08:33:46.055 INFO:teuthology.orchestra.run.smithi023.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2021-11-19T08:33:46.065 INFO:teuthology.orchestra.run.smithi023.stdout:(4/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 44 MB/s | 2.7 MB 00:00 2021-11-19T08:33:46.121 INFO:teuthology.orchestra.run.smithi023.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 375 kB/s | 34 kB 00:00 2021-11-19T08:33:46.195 INFO:teuthology.orchestra.run.smithi023.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2021-11-19T08:33:46.198 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:33:46.199 INFO:teuthology.orchestra.run.smithi023.stdout:Total 15 MB/s | 3.0 MB 00:00 2021-11-19T08:33:46.201 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:33:46.212 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:33:46.212 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:33:46.327 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:33:46.358 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:33:46.770 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2021-11-19T08:33:47.100 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:33:47.122 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-19T08:33:47.377 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-19T08:33:47.378 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2021-11-19T08:33:47.378 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2021-11-19T08:33:47.378 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2021-11-19T08:33:47.379 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-11-19T08:33:47.379 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-11-19T08:33:47.416 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-11-19T08:33:47.659 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-11-19T08:33:47.716 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-11-19T08:33:47.717 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:47.717 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:33:47.717 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2021-11-19T08:33:47.717 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2021-11-19T08:33:47.718 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2021-11-19T08:33:47.718 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-11-19T08:33:47.718 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-11-19T08:33:47.718 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.4.1-12.el8.noarch 2021-11-19T08:33:47.719 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:47.719 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:33:47.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2021-11-19T08:33:47.821 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-11-19T08:33:47.968 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2021-11-19T08:33:48.347 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:26 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:33:49.080 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2021-11-19T08:33:49.454 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-19T08:33:49.643 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:33:49.645 INFO:teuthology.orchestra.run.smithi134.stdout:===================================================================================================== 2021-11-19T08:33:49.645 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2021-11-19T08:33:49.645 INFO:teuthology.orchestra.run.smithi134.stdout:===================================================================================================== 2021-11-19T08:33:49.645 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:33:49.645 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2021-11-19T08:33:49.646 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2021-11-19T08:33:49.646 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 8.5.0-3.el8 CentOS-Base 643 k 2021-11-19T08:33:49.646 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 8.5.0-3.el8 CentOS-Base 170 k 2021-11-19T08:33:49.646 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2021-11-19T08:33:49.647 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2021-11-19T08:33:49.647 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-11-19T08:33:49.647 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-11-19T08:33:49.647 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2021-11-19T08:33:49.648 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:49.648 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:33:49.648 INFO:teuthology.orchestra.run.smithi134.stdout:===================================================================================================== 2021-11-19T08:33:49.648 INFO:teuthology.orchestra.run.smithi134.stdout:Install 8 Packages 2021-11-19T08:33:49.648 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:49.649 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 37 M 2021-11-19T08:33:49.649 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 211 M 2021-11-19T08:33:49.650 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:33:49.712 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2021-11-19T08:33:49.712 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2021-11-19T08:33:49.713 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2021-11-19T08:33:49.713 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2021-11-19T08:33:49.713 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-11-19T08:33:49.713 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-11-19T08:33:49.766 INFO:teuthology.orchestra.run.smithi134.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 72 MB/s | 8.6 MB 00:00 2021-11-19T08:33:49.808 INFO:teuthology.orchestra.run.smithi134.stdout:(2/8): openblas-0.3.15-3.el8.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2021-11-19T08:33:49.814 INFO:teuthology.orchestra.run.smithi134.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 39 MB/s | 225 kB 00:00 2021-11-19T08:33:49.899 INFO:teuthology.orchestra.run.smithi134.stdout:(4/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 19 MB/s | 4.9 MB 00:00 2021-11-19T08:33:49.923 INFO:teuthology.orchestra.run.smithi134.stdout:(5/8): libgfortran-8.5.0-3.el8.x86_64.rpm 26 MB/s | 643 kB 00:00 2021-11-19T08:33:49.935 INFO:teuthology.orchestra.run.smithi134.stdout:(6/8): libquadmath-8.5.0-3.el8.x86_64.rpm 14 MB/s | 170 kB 00:00 2021-11-19T08:33:50.103 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-11-19T08:33:50.103 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:50.104 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:33:50.104 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2021-11-19T08:33:50.104 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2021-11-19T08:33:50.104 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2021-11-19T08:33:50.105 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-11-19T08:33:50.105 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-11-19T08:33:50.105 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes-2.4.1-12.el8.noarch 2021-11-19T08:33:50.105 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:50.106 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:33:50.113 INFO:teuthology.orchestra.run.smithi134.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 46 MB/s | 14 MB 00:00 2021-11-19T08:33:50.142 INFO:teuthology.orchestra.run.smithi134.stdout:(8/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 10 MB/s | 3.7 MB 00:00 2021-11-19T08:33:50.146 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:33:50.146 INFO:teuthology.orchestra.run.smithi134.stdout:Total 73 MB/s | 37 MB 00:00 2021-11-19T08:33:50.161 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:33:50.179 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:33:50.180 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:33:50.190 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-mgr-diskprediction-local 2021-11-19T08:33:50.481 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:33:50.485 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:33:50.760 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:27 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:33:51.055 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:33:51.178 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-19T08:33:51.320 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-19T08:33:51.487 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-19T08:33:52.119 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-19T08:33:52.146 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:33:52.147 INFO:teuthology.orchestra.run.smithi023.stdout:===================================================================================================== 2021-11-19T08:33:52.148 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2021-11-19T08:33:52.148 INFO:teuthology.orchestra.run.smithi023.stdout:===================================================================================================== 2021-11-19T08:33:52.148 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:33:52.148 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2021-11-19T08:33:52.149 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2021-11-19T08:33:52.149 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran x86_64 8.5.0-3.el8 CentOS-Base 643 k 2021-11-19T08:33:52.149 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath x86_64 8.5.0-3.el8 CentOS-Base 170 k 2021-11-19T08:33:52.150 INFO:teuthology.orchestra.run.smithi023.stdout: openblas x86_64 0.3.15-3.el8 CentOS-AppStream 4.7 M 2021-11-19T08:33:52.150 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-threads x86_64 0.3.15-3.el8 CentOS-AppStream 4.9 M 2021-11-19T08:33:52.150 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-11-19T08:33:52.150 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-11-19T08:33:52.151 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2021-11-19T08:33:52.151 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:52.152 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:33:52.152 INFO:teuthology.orchestra.run.smithi023.stdout:===================================================================================================== 2021-11-19T08:33:52.152 INFO:teuthology.orchestra.run.smithi023.stdout:Install 8 Packages 2021-11-19T08:33:52.152 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:33:52.154 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 37 M 2021-11-19T08:33:52.154 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 211 M 2021-11-19T08:33:52.154 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:33:52.274 INFO:teuthology.orchestra.run.smithi023.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 70 MB/s | 8.6 MB 00:00 2021-11-19T08:33:52.298 INFO:teuthology.orchestra.run.smithi023.stdout:(2/8): openblas-0.3.15-3.el8.x86_64.rpm 32 MB/s | 4.7 MB 00:00 2021-11-19T08:33:52.304 INFO:teuthology.orchestra.run.smithi023.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 34 MB/s | 225 kB 00:00 2021-11-19T08:33:52.380 INFO:teuthology.orchestra.run.smithi023.stdout:(4/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 21 MB/s | 4.9 MB 00:00 2021-11-19T08:33:52.404 INFO:teuthology.orchestra.run.smithi023.stdout:(5/8): libgfortran-8.5.0-3.el8.x86_64.rpm 27 MB/s | 643 kB 00:00 2021-11-19T08:33:52.412 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2021-11-19T08:33:52.417 INFO:teuthology.orchestra.run.smithi023.stdout:(6/8): libquadmath-8.5.0-3.el8.x86_64.rpm 13 MB/s | 170 kB 00:00 2021-11-19T08:33:52.475 INFO:teuthology.orchestra.run.smithi023.stdout:(7/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2021-11-19T08:33:52.629 INFO:teuthology.orchestra.run.smithi023.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 42 MB/s | 14 MB 00:00 2021-11-19T08:33:52.633 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:33:52.634 INFO:teuthology.orchestra.run.smithi023.stdout:Total 76 MB/s | 37 MB 00:00 2021-11-19T08:33:52.649 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:33:52.671 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:33:52.671 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:33:52.991 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:33:53.000 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:33:53.030 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2021-11-19T08:33:53.186 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-19T08:33:53.547 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:33:53.701 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-19T08:33:53.836 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libquadmath-8.5.0-3.el8.x86_64 1/8 2021-11-19T08:33:53.923 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-19T08:33:54.017 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-19T08:33:54.295 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-11-19T08:33:54.553 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-19T08:33:54.636 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libgfortran-8.5.0-3.el8.x86_64 2/8 2021-11-19T08:33:54.801 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-0.3.15-3.el8.x86_64 3/8 2021-11-19T08:33:55.439 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 3/8 2021-11-19T08:33:55.626 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-19T08:33:56.360 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-19T08:33:56.413 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 4/8 2021-11-19T08:33:56.772 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-11-19T08:33:57.042 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-19T08:33:57.732 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2021-11-19T08:33:58.060 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-19T08:33:58.356 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-19T08:33:58.356 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2021-11-19T08:33:58.356 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2021-11-19T08:33:58.356 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2021-11-19T08:33:58.357 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-11-19T08:33:58.357 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-11-19T08:33:58.357 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2021-11-19T08:33:58.357 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-8.5.0-3.el8.x86_64 7/8 2021-11-19T08:33:58.707 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-8.5.0-3.el8.x86_64 8/8 2021-11-19T08:33:58.707 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:58.707 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:33:58.708 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2021-11-19T08:33:58.708 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-8.5.0-3.el8.x86_64 2021-11-19T08:33:58.708 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-8.5.0-3.el8.x86_64 2021-11-19T08:33:58.708 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.15-3.el8.x86_64 2021-11-19T08:33:58.709 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-threads-0.3.15-3.el8.x86_64 2021-11-19T08:33:58.709 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-11-19T08:33:58.709 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-11-19T08:33:58.709 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2021-11-19T08:33:58.709 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:33:58.710 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:33:58.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2021-11-19T08:33:58.972 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-11-19T08:33:59.414 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:37 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:00.328 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2021-11-19T08:34:00.491 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-19T08:34:00.722 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:00.724 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:00.724 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2021-11-19T08:34:00.724 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:00.724 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:34:00.725 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2021-11-19T08:34:00.725 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2021-11-19T08:34:00.725 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-11-19T08:34:00.725 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-11-19T08:34:00.725 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-11-19T08:34:00.726 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-11-19T08:34:00.726 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-11-19T08:34:00.726 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-11-19T08:34:00.726 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-11-19T08:34:00.727 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-11-19T08:34:00.727 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-11-19T08:34:00.727 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-11-19T08:34:00.727 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.7.2-1.el8 epel 61 k 2021-11-19T08:34:00.728 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-11-19T08:34:00.728 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:00.728 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:34:00.728 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:00.728 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2021-11-19T08:34:00.729 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:00.729 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 1.9 M 2021-11-19T08:34:00.730 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 24 M 2021-11-19T08:34:00.730 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:34:00.750 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 1.2 MB/s | 27 kB 00:00 2021-11-19T08:34:00.767 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 503 kB/s | 20 kB 00:00 2021-11-19T08:34:00.769 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 46 MB/s | 110 kB 00:00 2021-11-19T08:34:00.771 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 607 kB/s | 27 kB 00:00 2021-11-19T08:34:00.773 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 40 MB/s | 155 kB 00:00 2021-11-19T08:34:00.774 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 15 MB/s | 43 kB 00:00 2021-11-19T08:34:00.783 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.7 MB/s | 126 kB 00:00 2021-11-19T08:34:00.790 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2021-11-19T08:34:00.791 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2021-11-19T08:34:00.791 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 2/8 2021-11-19T08:34:00.791 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 3/8 2021-11-19T08:34:00.791 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-11-19T08:34:00.791 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-11-19T08:34:00.792 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2021-11-19T08:34:00.792 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libgfortran-8.5.0-3.el8.x86_64 7/8 2021-11-19T08:34:00.857 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 196 kB/s | 16 kB 00:00 2021-11-19T08:34:00.858 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 393 kB/s | 33 kB 00:00 2021-11-19T08:34:00.894 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 734 kB/s | 81 kB 00:00 2021-11-19T08:34:00.913 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-rsa-4.7.2-1.el8.noarch.rpm 1.1 MB/s | 61 kB 00:00 2021-11-19T08:34:00.923 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 2.1 MB/s | 61 kB 00:00 2021-11-19T08:34:01.056 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.9 MB/s | 1.2 MB 00:00 2021-11-19T08:34:01.059 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:01.060 INFO:teuthology.orchestra.run.smithi134.stdout:Total 5.7 MB/s | 1.9 MB 00:00 2021-11-19T08:34:01.069 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:34:01.084 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:34:01.084 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:34:01.155 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libquadmath-8.5.0-3.el8.x86_64 8/8 2021-11-19T08:34:01.155 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:01.156 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:34:01.156 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2021-11-19T08:34:01.157 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran-8.5.0-3.el8.x86_64 2021-11-19T08:34:01.158 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath-8.5.0-3.el8.x86_64 2021-11-19T08:34:01.158 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-0.3.15-3.el8.x86_64 2021-11-19T08:34:01.158 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-threads-0.3.15-3.el8.x86_64 2021-11-19T08:34:01.159 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-11-19T08:34:01.159 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-11-19T08:34:01.160 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2021-11-19T08:34:01.160 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:01.161 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:01.162 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:34:01.164 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:34:01.257 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-mgr-rook 2021-11-19T08:34:01.668 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:34:01.824 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:38 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:01.873 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-11-19T08:34:02.072 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-11-19T08:34:02.331 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.7.2-1.el8.noarch 3/13 2021-11-19T08:34:02.576 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-11-19T08:34:02.853 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-11-19T08:34:03.150 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-11-19T08:34:03.215 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:03.216 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:03.216 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2021-11-19T08:34:03.216 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:03.217 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:34:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2021-11-19T08:34:03.217 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2021-11-19T08:34:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-11-19T08:34:03.217 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-11-19T08:34:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-11-19T08:34:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-11-19T08:34:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-11-19T08:34:03.218 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-11-19T08:34:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-11-19T08:34:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-11-19T08:34:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-11-19T08:34:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-11-19T08:34:03.219 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa noarch 4.7.2-1.el8 epel 61 k 2021-11-19T08:34:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-11-19T08:34:03.220 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:03.220 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:34:03.220 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:03.221 INFO:teuthology.orchestra.run.smithi023.stdout:Install 13 Packages 2021-11-19T08:34:03.221 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:03.222 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 1.9 M 2021-11-19T08:34:03.222 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 24 M 2021-11-19T08:34:03.222 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:34:03.238 INFO:teuthology.orchestra.run.smithi023.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 27 kB 00:00 2021-11-19T08:34:03.257 INFO:teuthology.orchestra.run.smithi023.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 701 kB/s | 27 kB 00:00 2021-11-19T08:34:03.260 INFO:teuthology.orchestra.run.smithi023.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 36 MB/s | 110 kB 00:00 2021-11-19T08:34:03.261 INFO:teuthology.orchestra.run.smithi023.stdout:(4/13): python3-jsonpointer-1.10-11.el8.noarch. 481 kB/s | 20 kB 00:00 2021-11-19T08:34:03.265 INFO:teuthology.orchestra.run.smithi023.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 31 MB/s | 155 kB 00:00 2021-11-19T08:34:03.266 INFO:teuthology.orchestra.run.smithi023.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 8.2 MB/s | 43 kB 00:00 2021-11-19T08:34:03.268 INFO:teuthology.orchestra.run.smithi023.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.2 MB/s | 126 kB 00:00 2021-11-19T08:34:03.382 INFO:teuthology.orchestra.run.smithi023.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 141 kB/s | 16 kB 00:00 2021-11-19T08:34:03.411 INFO:teuthology.orchestra.run.smithi023.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 227 kB/s | 33 kB 00:00 2021-11-19T08:34:03.446 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-11-19T08:34:03.468 INFO:teuthology.orchestra.run.smithi023.stdout:(10/13): python3-rsa-4.7.2-1.el8.noarch.rpm 1.1 MB/s | 61 kB 00:00 2021-11-19T08:34:03.498 INFO:teuthology.orchestra.run.smithi023.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 2.0 MB/s | 61 kB 00:00 2021-11-19T08:34:03.579 INFO:teuthology.orchestra.run.smithi023.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 5.9 MB/s | 1.2 MB 00:00 2021-11-19T08:34:03.590 INFO:teuthology.orchestra.run.smithi023.stdout:(13/13): python3-google-auth-1.1.1-10.el8.noarc 253 kB/s | 81 kB 00:00 2021-11-19T08:34:03.593 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:03.593 INFO:teuthology.orchestra.run.smithi023.stdout:Total 5.1 MB/s | 1.9 MB 00:00 2021-11-19T08:34:03.604 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:34:03.620 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:34:03.620 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:34:03.694 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-11-19T08:34:03.703 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:34:03.707 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:34:04.215 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:34:04.431 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-11-19T08:34:04.654 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-11-19T08:34:04.827 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-11-19T08:34:04.863 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rsa-4.7.2-1.el8.noarch 3/13 2021-11-19T08:34:05.065 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-11-19T08:34:05.100 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-11-19T08:34:05.267 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-11-19T08:34:05.291 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-11-19T08:34:05.451 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-11-19T08:34:05.523 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-11-19T08:34:05.618 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-19T08:34:05.822 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-11-19T08:34:05.884 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-19T08:34:05.885 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2021-11-19T08:34:05.885 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-11-19T08:34:05.886 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-11-19T08:34:05.886 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-11-19T08:34:05.886 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-11-19T08:34:05.887 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-11-19T08:34:05.887 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-11-19T08:34:05.887 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-11-19T08:34:05.887 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-11-19T08:34:05.888 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-11-19T08:34:05.888 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-11-19T08:34:05.888 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.7.2-1.el8.noarch 12/13 2021-11-19T08:34:06.054 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-11-19T08:34:06.281 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-11-19T08:34:06.281 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:06.281 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:34:06.282 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2021-11-19T08:34:06.282 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-11-19T08:34:06.282 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-11-19T08:34:06.282 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-11-19T08:34:06.283 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-11-19T08:34:06.283 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-11-19T08:34:06.283 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-11-19T08:34:06.283 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-11-19T08:34:06.284 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-11-19T08:34:06.284 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-11-19T08:34:06.284 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-11-19T08:34:06.284 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.7.2-1.el8.noarch 2021-11-19T08:34:06.285 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-11-19T08:34:06.285 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:06.286 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:06.400 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2021-11-19T08:34:06.937 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:44 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:07.158 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-11-19T08:34:07.457 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-11-19T08:34:07.674 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-11-19T08:34:07.858 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-11-19T08:34:08.041 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-19T08:34:08.336 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2021-11-19T08:34:08.336 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2021-11-19T08:34:08.337 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-11-19T08:34:08.337 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-11-19T08:34:08.337 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-11-19T08:34:08.337 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-11-19T08:34:08.337 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-11-19T08:34:08.338 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-11-19T08:34:08.338 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-11-19T08:34:08.338 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-11-19T08:34:08.338 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-11-19T08:34:08.339 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-11-19T08:34:08.339 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rsa-4.7.2-1.el8.noarch 12/13 2021-11-19T08:34:08.612 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:08.613 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:08.613 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2021-11-19T08:34:08.614 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:08.614 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:34:08.614 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2021-11-19T08:34:08.614 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2021-11-19T08:34:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-11-19T08:34:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2021-11-19T08:34:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-11-19T08:34:08.615 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2021-11-19T08:34:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2021-11-19T08:34:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:08.616 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:34:08.616 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:08.617 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2021-11-19T08:34:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:08.618 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 5.8 M 2021-11-19T08:34:08.618 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 24 M 2021-11-19T08:34:08.618 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:34:08.636 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 4.4 MB/s | 89 kB 00:00 2021-11-19T08:34:08.669 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 10 MB/s | 538 kB 00:00 2021-11-19T08:34:08.713 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 49 MB/s | 4.8 MB 00:00 2021-11-19T08:34:08.783 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 144 kB/s | 21 kB 00:00 2021-11-19T08:34:08.882 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 289 kB/s | 49 kB 00:00 2021-11-19T08:34:08.931 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.3 MB/s | 355 kB 00:00 2021-11-19T08:34:08.933 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:08.934 INFO:teuthology.orchestra.run.smithi134.stdout:Total 18 MB/s | 5.8 MB 00:00 2021-11-19T08:34:08.938 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:34:08.947 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:34:08.948 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:34:08.969 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-11-19T08:34:08.969 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:08.970 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:34:08.970 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2021-11-19T08:34:08.970 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-11-19T08:34:08.970 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-11-19T08:34:08.971 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-11-19T08:34:08.971 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-11-19T08:34:08.972 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-11-19T08:34:08.972 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-11-19T08:34:08.972 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-11-19T08:34:08.972 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-11-19T08:34:08.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-11-19T08:34:08.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-11-19T08:34:08.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa-4.7.2-1.el8.noarch 2021-11-19T08:34:08.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-11-19T08:34:08.973 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:08.974 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:09.033 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:34:09.036 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:34:09.114 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-mgr-cephadm 2021-11-19T08:34:09.396 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:34:09.647 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-11-19T08:34:09.686 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:46 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:09.881 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-11-19T08:34:10.822 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2021-11-19T08:34:11.084 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:11.085 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:11.085 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2021-11-19T08:34:11.085 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:11.086 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:34:11.086 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2021-11-19T08:34:11.086 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2021-11-19T08:34:11.086 INFO:teuthology.orchestra.run.smithi023.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-11-19T08:34:11.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2021-11-19T08:34:11.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-11-19T08:34:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2021-11-19T08:34:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2021-11-19T08:34:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:11.088 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:34:11.089 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:11.089 INFO:teuthology.orchestra.run.smithi023.stdout:Install 6 Packages 2021-11-19T08:34:11.089 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:11.090 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 5.8 M 2021-11-19T08:34:11.090 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 24 M 2021-11-19T08:34:11.090 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:34:11.108 INFO:teuthology.orchestra.run.smithi023.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 4.3 MB/s | 89 kB 00:00 2021-11-19T08:34:11.144 INFO:teuthology.orchestra.run.smithi023.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 9.3 MB/s | 538 kB 00:00 2021-11-19T08:34:11.186 INFO:teuthology.orchestra.run.smithi023.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 49 MB/s | 4.8 MB 00:00 2021-11-19T08:34:11.195 INFO:teuthology.orchestra.run.smithi023.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 244 kB/s | 21 kB 00:00 2021-11-19T08:34:11.279 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2021-11-19T08:34:11.300 INFO:teuthology.orchestra.run.smithi023.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 430 kB/s | 49 kB 00:00 2021-11-19T08:34:11.307 INFO:teuthology.orchestra.run.smithi023.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.1 MB/s | 355 kB 00:00 2021-11-19T08:34:11.310 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:11.310 INFO:teuthology.orchestra.run.smithi023.stdout:Total 26 MB/s | 5.8 MB 00:00 2021-11-19T08:34:11.315 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:34:11.326 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:34:11.326 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:34:11.416 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:34:11.419 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:34:11.531 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2021-11-19T08:34:11.699 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:34:11.733 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-19T08:34:11.926 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-11-19T08:34:11.990 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-19T08:34:11.991 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2021-11-19T08:34:11.991 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2021-11-19T08:34:11.991 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2021-11-19T08:34:11.991 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-11-19T08:34:11.992 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-11-19T08:34:12.190 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-11-19T08:34:12.329 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2021-11-19T08:34:12.329 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:12.329 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:34:12.329 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2021-11-19T08:34:12.330 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2021-11-19T08:34:12.330 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2021-11-19T08:34:12.330 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:12.330 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:12.442 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2021-11-19T08:34:12.984 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:50 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:13.198 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2021-11-19T08:34:13.607 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2021-11-19T08:34:13.907 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2021-11-19T08:34:14.123 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-19T08:34:14.386 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2021-11-19T08:34:14.387 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2021-11-19T08:34:14.387 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2021-11-19T08:34:14.387 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2021-11-19T08:34:14.388 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-11-19T08:34:14.388 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-11-19T08:34:14.393 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:14.395 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:14.396 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2021-11-19T08:34:14.397 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:14.397 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:34:14.398 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2021-11-19T08:34:14.398 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2021-11-19T08:34:14.399 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-11-19T08:34:14.399 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:14.399 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:34:14.400 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:14.400 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2021-11-19T08:34:14.401 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:14.401 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 967 k 2021-11-19T08:34:14.402 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 3.2 M 2021-11-19T08:34:14.402 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:34:14.432 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.2 MB/s | 82 kB 00:00 2021-11-19T08:34:14.436 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 22 MB/s | 885 kB 00:00 2021-11-19T08:34:14.437 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:14.437 INFO:teuthology.orchestra.run.smithi134.stdout:Total 22 MB/s | 967 kB 00:00 2021-11-19T08:34:14.440 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:34:14.462 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:34:14.463 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:34:14.545 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:34:14.550 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:34:14.714 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:34:14.763 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2021-11-19T08:34:14.764 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:14.764 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:34:14.764 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2021-11-19T08:34:14.765 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2021-11-19T08:34:14.765 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2021-11-19T08:34:14.765 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:14.765 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:14.879 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-fuse 2021-11-19T08:34:15.075 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-11-19T08:34:15.283 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-19T08:34:15.449 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:52 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:15.672 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-19T08:34:15.672 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2021-11-19T08:34:16.021 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-11-19T08:34:16.021 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:16.021 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:34:16.022 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-11-19T08:34:16.022 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:16.023 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:16.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2021-11-19T08:34:16.806 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:54 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:16.868 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:16.870 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:16.871 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2021-11-19T08:34:16.871 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:16.872 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:34:16.872 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2021-11-19T08:34:16.872 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2021-11-19T08:34:16.873 INFO:teuthology.orchestra.run.smithi023.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-11-19T08:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:34:16.875 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:16.875 INFO:teuthology.orchestra.run.smithi023.stdout:Install 2 Packages 2021-11-19T08:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 967 k 2021-11-19T08:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 3.2 M 2021-11-19T08:34:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:34:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.5 MB/s | 82 kB 00:00 2021-11-19T08:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:(2/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 21 MB/s | 885 kB 00:00 2021-11-19T08:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:Total 22 MB/s | 967 kB 00:00 2021-11-19T08:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:34:17.028 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:34:17.031 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:34:17.209 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:34:17.537 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-11-19T08:34:17.773 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-19T08:34:18.137 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2021-11-19T08:34:18.137 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2021-11-19T08:34:18.138 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:18.139 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:18.139 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2021-11-19T08:34:18.139 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:18.139 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:34:18.140 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2021-11-19T08:34:18.140 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:18.141 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:34:18.142 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:18.142 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2021-11-19T08:34:18.143 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:18.144 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 131 k 2021-11-19T08:34:18.145 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 439 k 2021-11-19T08:34:18.145 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:34:18.153 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 10 MB/s | 131 kB 00:00 2021-11-19T08:34:18.154 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:18.155 INFO:teuthology.orchestra.run.smithi134.stdout:Total 9.2 MB/s | 131 kB 00:00 2021-11-19T08:34:18.155 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:34:18.161 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:34:18.161 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:34:18.211 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:34:18.214 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:34:18.329 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:34:18.500 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-11-19T08:34:18.501 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:18.501 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:34:18.501 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-11-19T08:34:18.501 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:18.502 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:18.548 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:18.690 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install librados-devel 2021-11-19T08:34:18.812 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:19.216 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:19.217 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:19.217 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:34:19.217 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2021-11-19T08:34:19.218 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:19.218 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:19.268 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:56 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:19.561 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2021-11-19T08:34:20.107 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:58 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:20.656 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:20.658 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:20.658 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2021-11-19T08:34:20.658 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:20.659 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:34:20.659 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2021-11-19T08:34:20.659 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:20.659 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:34:20.660 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:20.660 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2021-11-19T08:34:20.660 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:20.661 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 131 k 2021-11-19T08:34:20.661 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 439 k 2021-11-19T08:34:20.661 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:34:20.671 INFO:teuthology.orchestra.run.smithi023.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 10 MB/s | 131 kB 00:00 2021-11-19T08:34:20.672 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:20.672 INFO:teuthology.orchestra.run.smithi023.stdout:Total 9.0 MB/s | 131 kB 00:00 2021-11-19T08:34:20.673 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:34:20.680 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:34:20.680 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:34:20.736 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:34:20.740 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:34:20.865 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:34:21.088 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:34:21.090 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:21.359 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:21.372 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:21.374 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2021-11-19T08:34:21.374 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:21.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2021-11-19T08:34:21.723 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:21.723 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:21.724 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:34:21.724 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2021-11-19T08:34:21.724 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:21.724 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:21.899 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install libcephfs2 2021-11-19T08:34:22.004 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:00 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:22.471 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:59 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:23.294 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:23.295 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:23.296 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2021-11-19T08:34:23.296 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:23.296 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:34:23.296 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2021-11-19T08:34:23.297 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:23.297 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:34:23.297 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:23.297 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2021-11-19T08:34:23.298 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 24 k 2021-11-19T08:34:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 94 k 2021-11-19T08:34:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:34:23.307 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 24 kB 00:00 2021-11-19T08:34:23.308 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:23.309 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.8 MB/s | 24 kB 00:00 2021-11-19T08:34:23.309 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:34:23.313 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:34:23.313 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:34:23.315 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:34:23.318 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:34:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:34:23.565 INFO:teuthology.orchestra.run.smithi023.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:34:23.588 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:23.825 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:23.865 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:23.866 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2021-11-19T08:34:23.867 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:23.954 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install libcephfs-devel 2021-11-19T08:34:24.127 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:24.128 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:34:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2021-11-19T08:34:24.128 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:24.129 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:24.239 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2021-11-19T08:34:24.521 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:01 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:24.780 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:02 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:25.993 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:25.994 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:25.994 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2021-11-19T08:34:25.995 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:25.995 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:34:25.995 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2021-11-19T08:34:25.995 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:25.996 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:34:25.996 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:25.996 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2021-11-19T08:34:25.996 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:25.996 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 24 k 2021-11-19T08:34:25.997 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 94 k 2021-11-19T08:34:25.997 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:34:26.007 INFO:teuthology.orchestra.run.smithi023.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 24 kB 00:00 2021-11-19T08:34:26.007 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:26.008 INFO:teuthology.orchestra.run.smithi023.stdout:Total 1.8 MB/s | 24 kB 00:00 2021-11-19T08:34:26.008 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:34:26.012 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:34:26.013 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:34:26.015 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:34:26.018 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:34:26.090 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:34:26.120 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:34:26.254 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:26.408 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:26.410 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2021-11-19T08:34:26.410 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:26.500 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2021-11-19T08:34:26.502 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:26.842 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:26.843 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:26.843 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:34:26.844 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2021-11-19T08:34:26.844 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:26.845 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:26.949 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install librados2 2021-11-19T08:34:27.040 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:05 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:27.519 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:04 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:28.061 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:34:28.350 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:28.351 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2021-11-19T08:34:28.351 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:28.442 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2021-11-19T08:34:28.606 INFO:teuthology.orchestra.run.smithi023.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:34:28.909 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:28.910 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2021-11-19T08:34:28.911 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:28.981 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:06 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:28.996 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install librbd1 2021-11-19T08:34:29.566 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:06 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:29.994 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:34:30.287 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:30.288 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2021-11-19T08:34:30.288 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:30.375 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2021-11-19T08:34:30.913 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:08 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:31.773 INFO:teuthology.orchestra.run.smithi023.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:34:31.944 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:34:32.078 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:32.079 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2021-11-19T08:34:32.079 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:32.165 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install python3-rados 2021-11-19T08:34:32.235 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:32.236 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2021-11-19T08:34:32.237 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:32.324 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2021-11-19T08:34:32.734 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:09 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:32.864 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:10 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:33.839 INFO:teuthology.orchestra.run.smithi023.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:34:33.886 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:34:34.143 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:34.144 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2021-11-19T08:34:34.145 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:34.175 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:34.176 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2021-11-19T08:34:34.177 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:34.232 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install python3-rgw 2021-11-19T08:34:34.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2021-11-19T08:34:34.802 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:11 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:34.807 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:12 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:35.793 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:34:35.897 INFO:teuthology.orchestra.run.smithi023.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:34:36.083 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:36.084 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2021-11-19T08:34:36.085 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:36.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2021-11-19T08:34:36.203 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2021-11-19T08:34:36.204 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:36.282 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install python3-cephfs 2021-11-19T08:34:36.729 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:14 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:36.853 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:13 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:37.922 INFO:teuthology.orchestra.run.smithi023.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:34:38.077 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:38.078 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:38.078 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2021-11-19T08:34:38.079 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:38.079 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:34:38.079 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2021-11-19T08:34:38.079 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:38.079 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:34:38.080 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:38.080 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2021-11-19T08:34:38.080 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:38.080 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 89 k 2021-11-19T08:34:38.081 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 298 k 2021-11-19T08:34:38.081 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:34:38.090 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 7.3 MB/s | 89 kB 00:00 2021-11-19T08:34:38.091 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:38.092 INFO:teuthology.orchestra.run.smithi134.stdout:Total 6.6 MB/s | 89 kB 00:00 2021-11-19T08:34:38.092 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:34:38.103 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:34:38.104 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:34:38.155 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:34:38.158 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:34:38.228 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:38.230 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2021-11-19T08:34:38.230 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:38.281 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:34:38.306 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install python3-rbd 2021-11-19T08:34:38.511 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:38.772 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:38.876 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:15 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:39.091 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:39.091 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:39.092 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:34:39.092 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2021-11-19T08:34:39.093 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:39.093 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:39.317 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2021-11-19T08:34:39.862 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:17 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:39.997 INFO:teuthology.orchestra.run.smithi023.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2021-11-19T08:34:40.303 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:40.304 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2021-11-19T08:34:40.305 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:40.393 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install rbd-fuse 2021-11-19T08:34:40.962 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:17 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:41.152 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:41.153 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:41.154 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2021-11-19T08:34:41.154 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:41.154 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:34:41.155 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-19T08:34:41.155 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:41.155 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:34:41.156 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:41.156 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2021-11-19T08:34:41.156 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:41.157 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.7 M 2021-11-19T08:34:41.157 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 18 M 2021-11-19T08:34:41.157 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:34:41.215 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 61 MB/s | 3.7 MB 00:00 2021-11-19T08:34:41.216 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:41.217 INFO:teuthology.orchestra.run.smithi134.stdout:Total 59 MB/s | 3.7 MB 00:00 2021-11-19T08:34:41.217 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:34:41.231 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:34:41.231 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:34:41.295 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:34:41.298 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:34:41.762 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:34:41.962 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:42.326 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:42.327 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:42.327 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2021-11-19T08:34:42.327 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:42.328 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:34:42.328 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2021-11-19T08:34:42.328 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:42.328 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:34:42.328 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:42.329 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2021-11-19T08:34:42.329 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:42.329 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 89 k 2021-11-19T08:34:42.329 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 298 k 2021-11-19T08:34:42.330 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:34:42.339 INFO:teuthology.orchestra.run.smithi023.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 7.3 MB/s | 89 kB 00:00 2021-11-19T08:34:42.340 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:42.340 INFO:teuthology.orchestra.run.smithi023.stdout:Total 6.5 MB/s | 89 kB 00:00 2021-11-19T08:34:42.341 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:34:42.355 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:34:42.355 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:34:42.379 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:42.409 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:34:42.413 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:34:42.531 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:34:42.890 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:42.891 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:42.891 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:34:42.891 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2021-11-19T08:34:42.891 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:42.892 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:42.910 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:43.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2021-11-19T08:34:43.178 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:43.553 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:43.554 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:43.554 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:34:43.554 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2021-11-19T08:34:43.555 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:43.555 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:43.688 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:21 ago on Fri 19 Nov 2021 08:32:22 AM UTC. 2021-11-19T08:34:43.769 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install rbd-mirror 2021-11-19T08:34:44.337 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:21 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:44.993 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2021-11-19T08:34:44.994 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:44.994 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2021-11-19T08:34:44.994 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:44.995 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2021-11-19T08:34:44.995 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2021-11-19T08:34:44.995 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:44.995 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2021-11-19T08:34:44.995 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2021-11-19T08:34:44.996 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2021-11-19T08:34:44.996 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:44.996 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 173 k 2021-11-19T08:34:44.996 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 649 k 2021-11-19T08:34:44.997 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2021-11-19T08:34:45.007 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 13 MB/s | 173 kB 00:00 2021-11-19T08:34:45.008 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:45.009 INFO:teuthology.orchestra.run.smithi134.stdout:Total 12 MB/s | 173 kB 00:00 2021-11-19T08:34:45.009 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2021-11-19T08:34:45.022 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2021-11-19T08:34:45.022 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2021-11-19T08:34:45.072 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2021-11-19T08:34:45.075 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2021-11-19T08:34:45.187 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2021-11-19T08:34:45.452 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:45.711 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:45.736 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:45.738 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:45.738 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2021-11-19T08:34:45.738 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:45.738 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:34:45.739 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2021-11-19T08:34:45.739 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:45.739 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:34:45.739 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:45.740 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2021-11-19T08:34:45.740 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:45.740 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 3.7 M 2021-11-19T08:34:45.741 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 18 M 2021-11-19T08:34:45.742 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:34:45.795 INFO:teuthology.orchestra.run.smithi023.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 66 MB/s | 3.7 MB 00:00 2021-11-19T08:34:45.796 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:45.796 INFO:teuthology.orchestra.run.smithi023.stdout:Total 64 MB/s | 3.7 MB 00:00 2021-11-19T08:34:45.796 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:34:45.812 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:34:45.812 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:34:45.879 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:34:45.883 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:34:46.063 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:46.064 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:46.064 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2021-11-19T08:34:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2021-11-19T08:34:46.065 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:34:46.066 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2021-11-19T08:34:46.291 DEBUG:teuthology.parallel:result is None 2021-11-19T08:34:46.378 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:34:46.552 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:46.976 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:47.323 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:47.323 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:47.323 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:34:47.324 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2021-11-19T08:34:47.324 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:47.324 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:47.506 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install rbd-nbd 2021-11-19T08:34:48.079 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:25 ago on Fri 19 Nov 2021 08:32:23 AM UTC. 2021-11-19T08:34:49.515 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2021-11-19T08:34:49.515 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:49.516 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2021-11-19T08:34:49.516 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:49.516 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2021-11-19T08:34:49.516 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2021-11-19T08:34:49.516 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:49.517 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2021-11-19T08:34:49.517 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2021-11-19T08:34:49.517 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2021-11-19T08:34:49.517 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:49.518 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 173 k 2021-11-19T08:34:49.518 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 649 k 2021-11-19T08:34:49.518 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2021-11-19T08:34:49.544 INFO:teuthology.orchestra.run.smithi023.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 5.7 MB/s | 173 kB 00:00 2021-11-19T08:34:49.545 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2021-11-19T08:34:49.545 INFO:teuthology.orchestra.run.smithi023.stdout:Total 5.5 MB/s | 173 kB 00:00 2021-11-19T08:34:49.545 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2021-11-19T08:34:49.559 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2021-11-19T08:34:49.560 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2021-11-19T08:34:49.613 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2021-11-19T08:34:49.617 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2021-11-19T08:34:49.771 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2021-11-19T08:34:49.959 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:50.262 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:50.579 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2021-11-19T08:34:50.580 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:50.580 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2021-11-19T08:34:50.580 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2021-11-19T08:34:50.581 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:34:50.581 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2021-11-19T08:34:50.787 DEBUG:teuthology.parallel:result is None 2021-11-19T08:34:50.788 DEBUG:teuthology.orchestra.run.smithi023:> sudo lsb_release -is 2021-11-19T08:34:50.823 INFO:teuthology.orchestra.run.smithi023.stdout:CentOSStream 2021-11-19T08:34:50.825 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-19T08:34:50.825 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-19T08:34:50.826 INFO:teuthology.packaging:ref: None 2021-11-19T08:34:50.826 INFO:teuthology.packaging:tag: v16.2.4 2021-11-19T08:34:50.826 INFO:teuthology.packaging:branch: None 2021-11-19T08:34:50.827 INFO:teuthology.packaging:sha1: e77c7dee6c987c6680b57de9907bbc4d4962f2b1 2021-11-19T08:34:50.859 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-19T08:34:50.860 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-19T08:34:51.084 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-11-19T08:34:51.107 INFO:teuthology.orchestra.run.smithi023.stdout:16.2.4-0.el8 2021-11-19T08:34:51.108 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2021-11-19T08:34:51.109 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2021-11-19T08:34:51.109 DEBUG:teuthology.orchestra.run.smithi134:> sudo lsb_release -is 2021-11-19T08:34:51.151 INFO:teuthology.orchestra.run.smithi134.stdout:CentOSStream 2021-11-19T08:34:51.152 DEBUG:teuthology.misc:System to be installed: CentOSStream 2021-11-19T08:34:51.153 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2021-11-19T08:34:51.153 INFO:teuthology.packaging:ref: None 2021-11-19T08:34:51.154 INFO:teuthology.packaging:tag: v16.2.4 2021-11-19T08:34:51.154 INFO:teuthology.packaging:branch: None 2021-11-19T08:34:51.155 INFO:teuthology.packaging:sha1: e77c7dee6c987c6680b57de9907bbc4d4962f2b1 2021-11-19T08:34:51.185 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-19T08:34:51.187 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2021-11-19T08:34:51.423 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-11-19T08:34:51.449 INFO:teuthology.orchestra.run.smithi134.stdout:16.2.4-0.el8 2021-11-19T08:34:51.449 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2021-11-19T08:34:51.450 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2021-11-19T08:34:51.450 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-11-19T08:34:51.451 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2021-11-19T08:34:51.452 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-11-19T08:34:51.478 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2021-11-19T08:34:51.478 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-11-19T08:34:51.519 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-11-19T08:34:51.519 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2021-11-19T08:34:51.520 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/daemon-helper 2021-11-19T08:34:51.547 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-11-19T08:34:51.613 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2021-11-19T08:34:51.613 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2021-11-19T08:34:51.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-11-19T08:34:51.714 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-11-19T08:34:51.715 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2021-11-19T08:34:51.715 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/adjust-ulimits 2021-11-19T08:34:51.741 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-11-19T08:34:51.807 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2021-11-19T08:34:51.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2021-11-19T08:34:51.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-11-19T08:34:51.903 INFO:teuthology.run_tasks:Running task cephadm... 2021-11-19T08:34:51.987 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'e77c7dee6c987c6680b57de9907bbc4d4962f2b1'} 2021-11-19T08:34:51.988 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2021-11-19T08:34:51.989 INFO:tasks.cephadm:Cluster fsid is 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:34:51.989 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2021-11-19T08:34:51.990 INFO:tasks.cephadm:No mon roles; fabricating mons 2021-11-19T08:34:51.990 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi023': '172.21.15.23', 'mon.smithi134': '172.21.15.134'} 2021-11-19T08:34:51.991 INFO:tasks.cephadm:Normalizing hostnames... 2021-11-19T08:34:51.991 DEBUG:teuthology.orchestra.run.smithi023:> sudo hostname $(hostname -s) 2021-11-19T08:34:52.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo hostname $(hostname -s) 2021-11-19T08:34:52.048 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2021-11-19T08:34:52.049 DEBUG:teuthology.orchestra.run.smithi023:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2021-11-19T08:34:52.311 INFO:teuthology.orchestra.run.smithi023.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 19 08:34 /home/ubuntu/cephtest/cephadm 2021-11-19T08:34:52.312 DEBUG:teuthology.orchestra.run.smithi134:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2021-11-19T08:34:52.415 INFO:teuthology.orchestra.run.smithi134.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 19 08:34 /home/ubuntu/cephtest/cephadm 2021-11-19T08:34:52.416 DEBUG:teuthology.orchestra.run.smithi023:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-11-19T08:34:52.433 DEBUG:teuthology.orchestra.run.smithi134:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-11-19T08:34:52.502 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph 2021-11-19T08:34:52.529 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph 2021-11-19T08:34:52.556 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 777 /etc/ceph 2021-11-19T08:34:52.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /etc/ceph 2021-11-19T08:34:52.622 INFO:tasks.cephadm:Writing seed config... 2021-11-19T08:34:52.624 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2021-11-19T08:34:52.624 INFO:tasks.cephadm: override: [client] debug client = 20 2021-11-19T08:34:52.625 INFO:tasks.cephadm: override: [client] debug ms = 1 2021-11-19T08:34:52.625 INFO:tasks.cephadm: override: [client] rados mon op timeout = 15m 2021-11-19T08:34:52.625 INFO:tasks.cephadm: override: [client] rados osd op timeout = 15m 2021-11-19T08:34:52.626 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2021-11-19T08:34:52.626 INFO:tasks.cephadm: override: [mds] debug mds = 20 2021-11-19T08:34:52.626 INFO:tasks.cephadm: override: [mds] debug ms = 1 2021-11-19T08:34:52.627 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2021-11-19T08:34:52.627 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2021-11-19T08:34:52.627 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2021-11-19T08:34:52.628 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2021-11-19T08:34:52.628 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2021-11-19T08:34:52.628 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 15m 2021-11-19T08:34:52.629 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 15m 2021-11-19T08:34:52.629 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2021-11-19T08:34:52.629 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2021-11-19T08:34:52.630 INFO:tasks.cephadm: override: [mon] debug mon = 20 2021-11-19T08:34:52.630 INFO:tasks.cephadm: override: [mon] debug ms = 1 2021-11-19T08:34:52.630 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2021-11-19T08:34:52.631 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2021-11-19T08:34:52.631 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2021-11-19T08:34:52.632 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2021-11-19T08:34:52.633 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2021-11-19T08:34:52.633 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2021-11-19T08:34:52.634 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2021-11-19T08:34:52.634 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2021-11-19T08:34:52.634 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2021-11-19T08:34:52.635 INFO:tasks.cephadm: override: [osd] debug ms = 1 2021-11-19T08:34:52.635 INFO:tasks.cephadm: override: [osd] debug osd = 20 2021-11-19T08:34:52.635 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2021-11-19T08:34:52.636 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2021-11-19T08:34:52.636 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2021-11-19T08:34:52.637 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2021-11-19T08:34:52.637 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2021-11-19T08:34:52.638 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2021-11-19T08:34:52.638 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2021-11-19T08:34:52.640 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2021-11-19T08:34:52.641 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2021-11-19T08:34:52.657 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 908b41ee-4913-11ec-8c2c-001a4aab830c mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 15m rados osd op timeout = 15m [mds] debug mds = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 15m rados osd op timeout = 15m 2021-11-19T08:34:52.657 DEBUG:teuthology.orchestra.run.smithi023:mon.smithi023> sudo journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi023.service 2021-11-19T08:34:52.700 INFO:tasks.cephadm:Bootstrapping... 2021-11-19T08:34:52.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.23 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2021-11-19T08:34:52.724 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:-- Logs begin at Fri 2021-11-19 08:24:48 UTC. -- 2021-11-19T08:34:52.957 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-19T08:34:53.800 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: 3.4.1-dev 2021-11-19T08:34:53.841 INFO:teuthology.orchestra.run.smithi023.stderr:Verifying podman|docker is present... 2021-11-19T08:34:53.841 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-19T08:34:53.883 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: 3.4.1-dev 2021-11-19T08:34:53.982 INFO:teuthology.orchestra.run.smithi023.stderr:Verifying lvm2 is present... 2021-11-19T08:34:53.982 INFO:teuthology.orchestra.run.smithi023.stderr:Verifying time synchronization is in place... 2021-11-19T08:34:53.983 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled chrony.service 2021-11-19T08:34:53.989 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-11-19T08:34:53.990 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active chrony.service 2021-11-19T08:34:53.996 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: inactive 2021-11-19T08:34:53.996 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled chronyd.service 2021-11-19T08:34:54.002 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: enabled 2021-11-19T08:34:54.003 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active chronyd.service 2021-11-19T08:34:54.008 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: active 2021-11-19T08:34:54.009 INFO:teuthology.orchestra.run.smithi023.stderr:Unit chronyd.service is enabled and running 2021-11-19T08:34:54.009 INFO:teuthology.orchestra.run.smithi023.stderr:Repeating the final host check... 2021-11-19T08:34:54.009 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2021-11-19T08:34:54.047 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: 3.4.1-dev 2021-11-19T08:34:54.150 INFO:teuthology.orchestra.run.smithi023.stderr:podman|docker (/bin/podman) is present 2021-11-19T08:34:54.151 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl is present 2021-11-19T08:34:54.151 INFO:teuthology.orchestra.run.smithi023.stderr:lvcreate is present 2021-11-19T08:34:54.151 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled chrony.service 2021-11-19T08:34:54.156 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-11-19T08:34:54.156 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active chrony.service 2021-11-19T08:34:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: inactive 2021-11-19T08:34:54.162 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled chronyd.service 2021-11-19T08:34:54.168 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: enabled 2021-11-19T08:34:54.168 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active chronyd.service 2021-11-19T08:34:54.173 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: active 2021-11-19T08:34:54.175 INFO:teuthology.orchestra.run.smithi023.stderr:Unit chronyd.service is enabled and running 2021-11-19T08:34:54.175 INFO:teuthology.orchestra.run.smithi023.stderr:Host looks OK 2021-11-19T08:34:54.175 INFO:teuthology.orchestra.run.smithi023.stderr:Cluster fsid: 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:34:54.176 INFO:teuthology.orchestra.run.smithi023.stderr:Acquiring lock 140595876055528 on /run/cephadm/908b41ee-4913-11ec-8c2c-001a4aab830c.lock 2021-11-19T08:34:54.176 INFO:teuthology.orchestra.run.smithi023.stderr:Lock 140595876055528 acquired on /run/cephadm/908b41ee-4913-11ec-8c2c-001a4aab830c.lock 2021-11-19T08:34:54.176 INFO:teuthology.orchestra.run.smithi023.stderr:Verifying IP 172.21.15.23 port 3300 ... 2021-11-19T08:34:54.176 INFO:teuthology.orchestra.run.smithi023.stderr:Verifying IP 172.21.15.23 port 6789 ... 2021-11-19T08:34:54.177 INFO:teuthology.orchestra.run.smithi023.stderr:Base mon IP is 172.21.15.23, final addrv is [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2021-11-19T08:34:54.177 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /sbin/ip route ls 2021-11-19T08:34:54.178 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2021-11-19T08:34:54.178 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.23 metric 100 2021-11-19T08:34:54.179 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /sbin/ip -6 route ls 2021-11-19T08:34:54.181 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2021-11-19T08:34:54.182 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2021-11-19T08:34:54.182 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2021-11-19T08:34:54.182 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /sbin/ip -6 addr ls 2021-11-19T08:34:54.183 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2021-11-19T08:34:54.184 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: inet6 ::1/128 scope host 2021-11-19T08:34:54.184 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-11-19T08:34:54.184 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2021-11-19T08:34:54.184 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link noprefixroute 2021-11-19T08:34:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-11-19T08:34:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:Mon IP 172.21.15.23 is in CIDR network 172.21.0.0/20 2021-11-19T08:34:54.186 INFO:teuthology.orchestra.run.smithi023.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2021-11-19T08:34:54.186 INFO:teuthology.orchestra.run.smithi023.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2021-11-19T08:34:54.186 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2021-11-19T08:34:54.255 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2021-11-19T08:34:54.667 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Getting image source signatures 2021-11-19T08:34:54.681 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2021-11-19T08:34:54.682 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2021-11-19T08:34:54.811 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2021-11-19T08:34:54.861 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2021-11-19T08:35:07.426 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2021-11-19T08:35:07.509 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Writing manifest to image destination 2021-11-19T08:35:07.510 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Storing signatures 2021-11-19T08:35:07.902 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2021-11-19T08:35:07.958 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2021-11-19T08:35:08.591 INFO:teuthology.orchestra.run.smithi023.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2021-11-19T08:35:08.999 INFO:teuthology.orchestra.run.smithi023.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2021-11-19T08:35:09.000 INFO:teuthology.orchestra.run.smithi023.stderr:Extracting ceph user uid/gid from container image... 2021-11-19T08:35:09.000 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2021-11-19T08:35:09.442 INFO:teuthology.orchestra.run.smithi023.stderr:stat: 167 167 2021-11-19T08:35:09.944 INFO:teuthology.orchestra.run.smithi023.stderr:Creating initial keys... 2021-11-19T08:35:09.945 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-19T08:35:10.301 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-authtool: AQC+YZdh00/iERAAp0fcWKW/ulza6drjKfQYeg== 2021-11-19T08:35:10.687 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-19T08:35:11.082 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-authtool: AQC/YZdhHgehBBAACxqyOC5Lx47VEPE1rIKY+w== 2021-11-19T08:35:11.432 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2021-11-19T08:35:11.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-authtool: AQC/YZdhMWcqMBAASQ/UxPXkl+fr1krHe3/T4w== 2021-11-19T08:35:12.279 INFO:teuthology.orchestra.run.smithi023.stderr:Creating initial monmap... 2021-11-19T08:35:12.279 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpi92rv1ng:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c --addv smithi023 [v2:172.21.15.23:3300,v1:172.21.15.23:6789] /tmp/monmap 2021-11-19T08:35:12.684 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2021-11-19T08:35:12.685 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:35:12.685 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-11-19T08:35:13.132 INFO:teuthology.orchestra.run.smithi023.stderr:monmaptool for smithi023 [v2:172.21.15.23:3300,v1:172.21.15.23:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2021-11-19T08:35:13.132 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/monmaptool: set fsid to 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:35:13.133 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-11-19T08:35:13.133 INFO:teuthology.orchestra.run.smithi023.stderr: 2021-11-19T08:35:13.133 INFO:teuthology.orchestra.run.smithi023.stderr:Creating mon... 2021-11-19T08:35:13.134 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023:/var/lib/ceph/mon/ceph-smithi023:z -v /tmp/ceph-tmp3i8k3wg8:/tmp/keyring:z -v /tmp/ceph-tmpi92rv1ng:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi023 --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2021-11-19T08:35:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.562+0000 7fc9b1f77700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-19T08:35:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.562+0000 7fc9b1f77700 1 imported monmap: 2021-11-19T08:35:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: epoch 0 2021-11-19T08:35:13.606 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: fsid 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:35:13.607 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: last_changed 2021-11-19T08:35:12.680933+0000 2021-11-19T08:35:13.607 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: created 2021-11-19T08:35:12.680933+0000 2021-11-19T08:35:13.607 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2021-11-19T08:35:13.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: election_strategy: 1 2021-11-19T08:35:13.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.smithi023 2021-11-19T08:35:13.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.562+0000 7fc9b1f77700 0 /usr/bin/ceph-mon: set fsid to 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:35:13.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: RocksDB version: 6.8.1 2021-11-19T08:35:13.609 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.609 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-19T08:35:13.609 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Compile date May 13 2021 2021-11-19T08:35:13.609 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: DB SUMMARY 2021-11-19T08:35:13.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi023/store.db dir, Total Num: 0, files: 2021-11-19T08:35:13.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi023/store.db: 2021-11-19T08:35:13.611 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.611 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.error_if_exists: 0 2021-11-19T08:35:13.611 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.create_if_missing: 1 2021-11-19T08:35:13.611 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.paranoid_checks: 1 2021-11-19T08:35:13.612 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.env: 0x5560cc4391c0 2021-11-19T08:35:13.612 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.fs: Posix File System 2021-11-19T08:35:13.612 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.info_log: 0x5560cdf0cec0 2021-11-19T08:35:13.612 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-19T08:35:13.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.statistics: (nil) 2021-11-19T08:35:13.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.use_fsync: 0 2021-11-19T08:35:13.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.max_log_file_size: 0 2021-11-19T08:35:13.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-19T08:35:13.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-19T08:35:13.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-19T08:35:13.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-19T08:35:13.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.allow_fallocate: 1 2021-11-19T08:35:13.615 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-19T08:35:13.615 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-19T08:35:13.615 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.use_direct_reads: 0 2021-11-19T08:35:13.615 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-19T08:35:13.616 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-19T08:35:13.616 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.db_log_dir: 2021-11-19T08:35:13.616 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi023/store.db 2021-11-19T08:35:13.616 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-19T08:35:13.616 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.max_subcompactions: 1 2021-11-19T08:35:13.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.max_background_flushes: -1 2021-11-19T08:35:13.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-19T08:35:13.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-19T08:35:13.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-19T08:35:13.618 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-19T08:35:13.618 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-19T08:35:13.618 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-19T08:35:13.618 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-19T08:35:13.618 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.write_buffer_manager: 0x5560cdf13530 2021-11-19T08:35:13.619 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-19T08:35:13.619 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-19T08:35:13.619 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-19T08:35:13.619 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-19T08:35:13.620 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-19T08:35:13.620 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-19T08:35:13.620 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-19T08:35:13.620 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-19T08:35:13.620 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-19T08:35:13.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.unordered_write: 0 2021-11-19T08:35:13.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-19T08:35:13.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-19T08:35:13.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-19T08:35:13.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-19T08:35:13.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.row_cache: None 2021-11-19T08:35:13.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.wal_filter: None 2021-11-19T08:35:13.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-19T08:35:13.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-19T08:35:13.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.preserve_deletes: 0 2021-11-19T08:35:13.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.two_write_queues: 0 2021-11-19T08:35:13.624 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-19T08:35:13.624 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.atomic_flush: 0 2021-11-19T08:35:13.624 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-19T08:35:13.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-19T08:35:13.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-19T08:35:13.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.log_readahead_size: 0 2021-11-19T08:35:13.626 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-19T08:35:13.626 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.max_background_jobs: 2 2021-11-19T08:35:13.626 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.max_background_compactions: -1 2021-11-19T08:35:13.627 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-19T08:35:13.627 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-19T08:35:13.627 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-19T08:35:13.627 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-19T08:35:13.628 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-19T08:35:13.628 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-19T08:35:13.628 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-19T08:35:13.628 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-19T08:35:13.628 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.max_open_files: -1 2021-11-19T08:35:13.629 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-19T08:35:13.629 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-19T08:35:13.629 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-19T08:35:13.629 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-19T08:35:13.630 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Compression algorithms supported: 2021-11-19T08:35:13.630 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-19T08:35:13.630 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: kZSTD supported: 0 2021-11-19T08:35:13.630 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: kXpressCompression supported: 0 2021-11-19T08:35:13.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-19T08:35:13.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: kLZ4Compression supported: 1 2021-11-19T08:35:13.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: kBZip2Compression supported: 0 2021-11-19T08:35:13.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: kZlibCompression supported: 1 2021-11-19T08:35:13.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: kSnappyCompression supported: 1 2021-11-19T08:35:13.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.563+0000 7fc9b1f77700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-19T08:35:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.565+0000 7fc9b1f77700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2021-11-19T08:35:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi023/store.db/MANIFEST-000001 2021-11-19T08:35:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-19T08:35:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-19T08:35:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.merge_operator: 2021-11-19T08:35:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compaction_filter: None 2021-11-19T08:35:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-19T08:35:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-19T08:35:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-19T08:35:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5560cde24120) 2021-11-19T08:35:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2021-11-19T08:35:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-19T08:35:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-19T08:35:13.637 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2021-11-19T08:35:13.637 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: index_type: 0 2021-11-19T08:35:13.637 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2021-11-19T08:35:13.638 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: index_shortening: 1 2021-11-19T08:35:13.638 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2021-11-19T08:35:13.638 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2021-11-19T08:35:13.638 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: checksum: 1 2021-11-19T08:35:13.639 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2021-11-19T08:35:13.639 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_cache: 0x5560cde5ad10 2021-11-19T08:35:13.639 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2021-11-19T08:35:13.639 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_cache_options: 2021-11-19T08:35:13.640 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: capacity : 536870912 2021-11-19T08:35:13.640 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2021-11-19T08:35:13.640 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2021-11-19T08:35:13.640 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2021-11-19T08:35:13.641 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2021-11-19T08:35:13.641 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2021-11-19T08:35:13.641 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_size: 4096 2021-11-19T08:35:13.641 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2021-11-19T08:35:13.641 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2021-11-19T08:35:13.642 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2021-11-19T08:35:13.642 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2021-11-19T08:35:13.642 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: partition_filters: 0 2021-11-19T08:35:13.642 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2021-11-19T08:35:13.642 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-19T08:35:13.643 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2021-11-19T08:35:13.643 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: verify_compression: 0 2021-11-19T08:35:13.643 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2021-11-19T08:35:13.643 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: format_version: 2 2021-11-19T08:35:13.644 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2021-11-19T08:35:13.644 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_align: 0 2021-11-19T08:35:13.644 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.644 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-19T08:35:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-19T08:35:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compression: NoCompression 2021-11-19T08:35:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-19T08:35:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-19T08:35:13.646 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-19T08:35:13.646 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.num_levels: 7 2021-11-19T08:35:13.646 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-19T08:35:13.646 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-19T08:35:13.647 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-19T08:35:13.647 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-19T08:35:13.647 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-19T08:35:13.647 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-19T08:35:13.647 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-19T08:35:13.648 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-19T08:35:13.648 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-19T08:35:13.648 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-19T08:35:13.648 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-19T08:35:13.649 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-19T08:35:13.649 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-19T08:35:13.649 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-19T08:35:13.649 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-19T08:35:13.649 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-19T08:35:13.650 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-19T08:35:13.650 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-19T08:35:13.650 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-19T08:35:13.650 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-19T08:35:13.651 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-19T08:35:13.651 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-19T08:35:13.651 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-19T08:35:13.651 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-19T08:35:13.652 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-19T08:35:13.652 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-19T08:35:13.652 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-19T08:35:13.652 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-19T08:35:13.653 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-19T08:35:13.653 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-19T08:35:13.653 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-19T08:35:13.653 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-19T08:35:13.653 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-19T08:35:13.654 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-19T08:35:13.654 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-19T08:35:13.654 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-19T08:35:13.654 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-19T08:35:13.654 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-19T08:35:13.655 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-19T08:35:13.655 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-19T08:35:13.655 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-19T08:35:13.655 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-19T08:35:13.656 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-19T08:35:13.656 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-19T08:35:13.656 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-19T08:35:13.656 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-19T08:35:13.656 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-19T08:35:13.657 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.table_properties_collectors: 2021-11-19T08:35:13.657 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.inplace_update_support: 0 2021-11-19T08:35:13.657 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-19T08:35:13.657 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-19T08:35:13.657 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-19T08:35:13.658 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-19T08:35:13.658 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.bloom_locality: 0 2021-11-19T08:35:13.658 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.max_successive_merges: 0 2021-11-19T08:35:13.658 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-19T08:35:13.659 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-19T08:35:13.659 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-19T08:35:13.659 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-19T08:35:13.659 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.ttl: 2592000 2021-11-19T08:35:13.660 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.568+0000 7fc9b1f77700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-19T08:35:13.660 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.569+0000 7fc9b1f77700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi023/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-19T08:35:13.660 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.660 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.569+0000 7fc9b1f77700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-11-19T08:35:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.572+0000 7fc9b1f77700 4 rocksdb: DB pointer 0x5560cdf1f800 2021-11-19T08:35:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.572+0000 7fc99ae10700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-19T08:35:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.572+0000 7fc99ae10700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-19T08:35:13.661 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2021-11-19T08:35:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-19T08:35:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-19T08:35:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-19T08:35:13.662 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-19T08:35:13.663 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-19T08:35:13.663 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-19T08:35:13.663 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-19T08:35:13.663 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.663 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-19T08:35:13.664 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-19T08:35:13.664 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-19T08:35:13.664 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-19T08:35:13.664 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-19T08:35:13.664 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.665 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-19T08:35:13.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-19T08:35:13.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-19T08:35:13.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-19T08:35:13.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-11-19T08:35:13.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-19T08:35:13.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-11-19T08:35:13.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-11-19T08:35:13.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-11-19T08:35:13.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-19T08:35:13.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-19T08:35:13.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-19T08:35:13.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-11-19T08:35:13.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-19T08:35:13.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-19T08:35:13.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-19T08:35:13.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-19T08:35:13.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-19T08:35:13.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-11-19T08:35:13.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-19T08:35:13.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-19T08:35:13.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-11-19T08:35:13.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-11-19T08:35:13.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-19T08:35:13.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-11-19T08:35:13.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-11-19T08:35:13.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-11-19T08:35:13.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-19T08:35:13.675 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-19T08:35:13.675 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-19T08:35:13.676 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.676 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-11-19T08:35:13.676 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2021-11-19T08:35:13.676 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.573+0000 7fc9b1f77700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2021-11-19T08:35:13.676 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.574+0000 7fc9b1f77700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2021-11-19T08:35:13.677 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2021-11-19T08:35:13.574+0000 7fc9b1f77700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi023 for mon.smithi023 2021-11-19T08:35:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:create mon.smithi023 on 2021-11-19T08:35:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:35:13.981 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl disable ceph.target 2021-11-19T08:35:13.988 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2021-11-19T08:35:14.073 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl enable ceph.target 2021-11-19T08:35:14.079 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2021-11-19T08:35:14.165 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl start ceph.target 2021-11-19T08:35:14.174 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl enable ceph-908b41ee-4913-11ec-8c2c-001a4aab830c.target 2021-11-19T08:35:14.178 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-908b41ee-4913-11ec-8c2c-001a4aab830c.target → /etc/systemd/system/ceph-908b41ee-4913-11ec-8c2c-001a4aab830c.target. 2021-11-19T08:35:14.178 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-908b41ee-4913-11ec-8c2c-001a4aab830c.target → /etc/systemd/system/ceph-908b41ee-4913-11ec-8c2c-001a4aab830c.target. 2021-11-19T08:35:14.259 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl start ceph-908b41ee-4913-11ec-8c2c-001a4aab830c.target 2021-11-19T08:35:14.267 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl daemon-reload 2021-11-19T08:35:14.351 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl stop ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi023 2021-11-19T08:35:14.377 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl reset-failed ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi023 2021-11-19T08:35:14.382 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Failed to reset failed state of unit ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi023.service: Unit ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi023.service not loaded. 2021-11-19T08:35:14.383 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl enable ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi023 2021-11-19T08:35:14.389 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Created symlink /etc/systemd/system/ceph-908b41ee-4913-11ec-8c2c-001a4aab830c.target.wants/ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi023.service → /etc/systemd/system/ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@.service. 2021-11-19T08:35:14.474 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl start ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi023 2021-11-19T08:35:14.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:14 smithi023 systemd[1]: Starting Ceph mon.smithi023 for 908b41ee-4913-11ec-8c2c-001a4aab830c... 2021-11-19T08:35:15.004 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:14 smithi023 podman[32758]: 2021-11-19T08:35:15.004 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:14 smithi023 bash[32681]: 69d0e12cf96c89b1850074fb2223c098e5ba5afb96e8215638682f6fa1f6ccb2 2021-11-19T08:35:15.004 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:14 smithi023 conmon[32798]: cluster 2021-11-19T08:35:14.953020+0000 mon.smithi023 (mon.0) 0 : 2021-11-19T08:35:15.005 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:14 smithi023 conmon[32798]: [INF] mkfs 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:35:15.005 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:14 smithi023 conmon[32798]: cluster 2021-11-19T08 2021-11-19T08:35:15.005 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:14 smithi023 conmon[32798]: :35:14.934633+0000 mon.smithi023 (mon.0) 1 : cluster [INF] mon.smithi023 is new leader, mons smithi023 in quorum (ranks 0) 2021-11-19T08:35:15.006 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-19T08:35:15.014 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: disabled 2021-11-19T08:35:15.015 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active firewalld.service 2021-11-19T08:35:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: inactive 2021-11-19T08:35:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:firewalld.service is not enabled 2021-11-19T08:35:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:Not possible to enable service . firewalld.service is not available 2021-11-19T08:35:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-11-19T08:35:15.024 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for mon to start... 2021-11-19T08:35:15.024 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for mon... 2021-11-19T08:35:15.024 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023:/var/lib/ceph/mon/ceph-smithi023:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2021-11-19T08:35:15.267 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:15 smithi023 systemd[1]: Started Ceph mon.smithi023 for 908b41ee-4913-11ec-8c2c-001a4aab830c. 2021-11-19T08:35:15.529 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.540+0000 7f4e7b149700 1 Processor -- start 2021-11-19T08:35:15.529 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.540+0000 7f4e7b149700 1 -- start start 2021-11-19T08:35:15.530 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.540+0000 7f4e7b149700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e7406fc30 0x7f4e740fe830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:15.530 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.540+0000 7f4e7b149700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e740fed20 con 0x7f4e7406fc30 2021-11-19T08:35:15.530 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.541+0000 7f4e78ee5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e7406fc30 0x7f4e740fe830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:15.530 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.541+0000 7f4e78ee5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e7406fc30 0x7f4e740fe830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45918/0 (socket says 172.21.15.23:45918) 2021-11-19T08:35:15.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.541+0000 7f4e78ee5700 1 -- 172.21.15.23:0/3967069591 learned_addr learned my addr 172.21.15.23:0/3967069591 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:15.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.541+0000 7f4e78ee5700 1 -- 172.21.15.23:0/3967069591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e740fee60 con 0x7f4e7406fc30 2021-11-19T08:35:15.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.541+0000 7f4e78ee5700 1 --2- 172.21.15.23:0/3967069591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e7406fc30 0x7f4e740fe830 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f4e70009960 tx=0x7f4e700092d0).ready entity=mon.0 client_cookie=29f2a87c4c5cd62b server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:15.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.541+0000 7f4e6b7fe700 1 -- 172.21.15.23:0/3967069591 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e7000b9c0 con 0x7f4e7406fc30 2021-11-19T08:35:15.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.541+0000 7f4e6b7fe700 1 -- 172.21.15.23:0/3967069591 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7000bb20 con 0x7f4e7406fc30 2021-11-19T08:35:15.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.541+0000 7f4e6b7fe700 1 -- 172.21.15.23:0/3967069591 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e7000bdf0 con 0x7f4e7406fc30 2021-11-19T08:35:15.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.542+0000 7f4e7b149700 1 -- 172.21.15.23:0/3967069591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e7406fc30 msgr2=0x7f4e740fe830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:15.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.542+0000 7f4e7b149700 1 --2- 172.21.15.23:0/3967069591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e7406fc30 0x7f4e740fe830 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f4e70009960 tx=0x7f4e700092d0).stop 2021-11-19T08:35:15.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.542+0000 7f4e7b149700 1 -- 172.21.15.23:0/3967069591 shutdown_connections 2021-11-19T08:35:15.535 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.542+0000 7f4e7b149700 1 --2- 172.21.15.23:0/3967069591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e7406fc30 0x7f4e740fe830 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:15.536 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.542+0000 7f4e7b149700 1 -- 172.21.15.23:0/3967069591 >> 172.21.15.23:0/3967069591 conn(0x7f4e740f6310 msgr2=0x7f4e740f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:15.536 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.542+0000 7f4e7b149700 1 -- 172.21.15.23:0/3967069591 shutdown_connections 2021-11-19T08:35:15.536 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.542+0000 7f4e7b149700 1 -- 172.21.15.23:0/3967069591 wait complete. 2021-11-19T08:35:15.536 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.543+0000 7f4e7b149700 1 Processor -- start 2021-11-19T08:35:15.537 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.543+0000 7f4e7b149700 1 -- start start 2021-11-19T08:35:15.537 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.544+0000 7f4e7b149700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e7406fc30 0x7f4e7410a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:15.537 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.544+0000 7f4e7b149700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7410a690 con 0x7f4e7406fc30 2021-11-19T08:35:15.537 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.544+0000 7f4e78ee5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e7406fc30 0x7f4e7410a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:15.537 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.544+0000 7f4e78ee5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e7406fc30 0x7f4e7410a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45920/0 (socket says 172.21.15.23:45920) 2021-11-19T08:35:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.544+0000 7f4e78ee5700 1 -- 172.21.15.23:0/3076981392 learned_addr learned my addr 172.21.15.23:0/3076981392 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.544+0000 7f4e78ee5700 1 -- 172.21.15.23:0/3076981392 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e70009570 con 0x7f4e7406fc30 2021-11-19T08:35:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.544+0000 7f4e78ee5700 1 --2- 172.21.15.23:0/3076981392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e7406fc30 0x7f4e7410a1a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4e70009f20 tx=0x7f4e70017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.544+0000 7f4e697fa700 1 -- 172.21.15.23:0/3076981392 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e7001c420 con 0x7f4e7406fc30 2021-11-19T08:35:15.539 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.544+0000 7f4e697fa700 1 -- 172.21.15.23:0/3076981392 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7001c580 con 0x7f4e7406fc30 2021-11-19T08:35:15.539 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.544+0000 7f4e7b149700 1 -- 172.21.15.23:0/3076981392 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e7410a890 con 0x7f4e7406fc30 2021-11-19T08:35:15.539 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.544+0000 7f4e7b149700 1 -- 172.21.15.23:0/3076981392 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7410aca0 con 0x7f4e7406fc30 2021-11-19T08:35:15.539 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.545+0000 7f4e697fa700 1 -- 172.21.15.23:0/3076981392 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e7001c850 con 0x7f4e7406fc30 2021-11-19T08:35:15.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.545+0000 7f4e697fa700 1 -- 172.21.15.23:0/3076981392 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f4e7002c070 con 0x7f4e7406fc30 2021-11-19T08:35:15.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.545+0000 7f4e697fa700 1 -- 172.21.15.23:0/3076981392 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4e70025630 con 0x7f4e7406fc30 2021-11-19T08:35:15.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.546+0000 7f4e7b149700 1 -- 172.21.15.23:0/3076981392 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e54004fa0 con 0x7f4e7406fc30 2021-11-19T08:35:15.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.548+0000 7f4e697fa700 1 -- 172.21.15.23:0/3076981392 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4e700449c0 con 0x7f4e7406fc30 2021-11-19T08:35:15.663 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.673+0000 7f4e7b149700 1 -- 172.21.15.23:0/3076981392 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f4e54005940 con 0x7f4e7406fc30 2021-11-19T08:35:15.664 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.673+0000 7f4e697fa700 1 -- 172.21.15.23:0/3076981392 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f4e70033030 con 0x7f4e7406fc30 2021-11-19T08:35:15.664 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: cluster: 2021-11-19T08:35:15.664 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: id: 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:35:15.665 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: health: HEALTH_OK 2021-11-19T08:35:15.665 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.665 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: services: 2021-11-19T08:35:15.665 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi023 (age 0.722781s) 2021-11-19T08:35:15.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mgr: no daemons active 2021-11-19T08:35:15.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2021-11-19T08:35:15.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: data: 2021-11-19T08:35:15.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2021-11-19T08:35:15.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2021-11-19T08:35:15.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2021-11-19T08:35:15.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: pgs: 2021-11-19T08:35:15.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.674+0000 7f4e7b149700 1 -- 172.21.15.23:0/3076981392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e7406fc30 msgr2=0x7f4e7410a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:15.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.674+0000 7f4e7b149700 1 --2- 172.21.15.23:0/3076981392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e7406fc30 0x7f4e7410a1a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4e70009f20 tx=0x7f4e70017000).stop 2021-11-19T08:35:15.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.674+0000 7f4e7b149700 1 -- 172.21.15.23:0/3076981392 shutdown_connections 2021-11-19T08:35:15.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.674+0000 7f4e7b149700 1 --2- 172.21.15.23:0/3076981392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e7406fc30 0x7f4e7410a1a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:15.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.674+0000 7f4e7b149700 1 -- 172.21.15.23:0/3076981392 >> 172.21.15.23:0/3076981392 conn(0x7f4e740f6310 msgr2=0x7f4e740f8060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:15.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.674+0000 7f4e7b149700 1 -- 172.21.15.23:0/3076981392 shutdown_connections 2021-11-19T08:35:15.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:15.674+0000 7f4e7b149700 1 -- 172.21.15.23:0/3076981392 wait complete. 2021-11-19T08:35:16.063 INFO:teuthology.orchestra.run.smithi023.stderr:mon is available 2021-11-19T08:35:16.063 INFO:teuthology.orchestra.run.smithi023.stderr:Assimilating anything we can from ceph.conf... 2021-11-19T08:35:16.063 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z -v /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023:/var/lib/ceph/mon/ceph-smithi023:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi023/config 2021-11-19T08:35:16.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:15 smithi023 conmon[32798]: cluster 2021-11-19T08:35:14.952460+0000 mon.smithi023 (mon.0) 2 : cluster [INF] mon.smithi023 is new leader, mons smithi023 in quorum (ranks 0) 2021-11-19T08:35:16.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:15 smithi023 conmon[32798]: cluster 2021-11-19T08:35:14.952589+0000 mon.smithi023 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi023=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} 2021-11-19T08:35:16.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:15 smithi023 conmon[32798]: cluster 2021-11-19T08:35:14.954327+0000 mon.smithi023 (mon.0) 4 : cluster [DBG] fsmap 2021-11-19T08:35:16.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:15 smithi023 conmon[32798]: cluster 2021-11-19T08:35:14.958158+0000 mon.smithi023 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-11-19T08:35:16.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:15 smithi023 conmon[32798]: cluster 2021-11-19T08:35:14.959258+0000 mon.smithi023 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2021-11-19T08:35:16.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:15 smithi023 conmon[32798]: audit 2021-11-19T08:35:15.675211+0000 mon.smithi023 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.23:0/3076981392' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-11-19T08:35:16.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.563+0000 7fee8db75700 1 Processor -- start 2021-11-19T08:35:16.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.563+0000 7fee8db75700 1 -- start start 2021-11-19T08:35:16.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.563+0000 7fee8db75700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee880fd280 0x7fee880fd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:16.555 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.563+0000 7fee8db75700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee880fdb90 con 0x7fee880fd280 2021-11-19T08:35:16.555 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.563+0000 7fee877fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee880fd280 0x7fee880fd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:16.555 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.563+0000 7fee877fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee880fd280 0x7fee880fd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45922/0 (socket says 172.21.15.23:45922) 2021-11-19T08:35:16.555 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.563+0000 7fee877fe700 1 -- 172.21.15.23:0/280362569 learned_addr learned my addr 172.21.15.23:0/280362569 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:16.556 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.564+0000 7fee877fe700 1 -- 172.21.15.23:0/280362569 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee880fdcd0 con 0x7fee880fd280 2021-11-19T08:35:16.556 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.564+0000 7fee877fe700 1 --2- 172.21.15.23:0/280362569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee880fd280 0x7fee880fd6a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fee7800b5f0 tx=0x7fee78006ee0).ready entity=mon.0 client_cookie=733d7e79c780bbb8 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:16.556 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.564+0000 7fee867fc700 1 -- 172.21.15.23:0/280362569 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee7800d600 con 0x7fee880fd280 2021-11-19T08:35:16.556 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.564+0000 7fee867fc700 1 -- 172.21.15.23:0/280362569 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7800d760 con 0x7fee880fd280 2021-11-19T08:35:16.557 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.564+0000 7fee867fc700 1 -- 172.21.15.23:0/280362569 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee7800da30 con 0x7fee880fd280 2021-11-19T08:35:16.557 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.565+0000 7fee8db75700 1 -- 172.21.15.23:0/280362569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee880fd280 msgr2=0x7fee880fd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:16.557 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.565+0000 7fee8db75700 1 --2- 172.21.15.23:0/280362569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee880fd280 0x7fee880fd6a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fee7800b5f0 tx=0x7fee78006ee0).stop 2021-11-19T08:35:16.557 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.565+0000 7fee8db75700 1 -- 172.21.15.23:0/280362569 shutdown_connections 2021-11-19T08:35:16.557 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.565+0000 7fee8db75700 1 --2- 172.21.15.23:0/280362569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee880fd280 0x7fee880fd6a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.565+0000 7fee8db75700 1 -- 172.21.15.23:0/280362569 >> 172.21.15.23:0/280362569 conn(0x7fee880f5a80 msgr2=0x7fee880f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.565+0000 7fee8db75700 1 -- 172.21.15.23:0/280362569 shutdown_connections 2021-11-19T08:35:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.565+0000 7fee8db75700 1 -- 172.21.15.23:0/280362569 wait complete. 2021-11-19T08:35:16.558 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.565+0000 7fee8db75700 1 Processor -- start 2021-11-19T08:35:16.559 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.565+0000 7fee8db75700 1 -- start start 2021-11-19T08:35:16.559 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.566+0000 7fee8db75700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee880fd280 0x7fee8810a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:16.559 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.566+0000 7fee8db75700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8810a6b0 con 0x7fee880fd280 2021-11-19T08:35:16.559 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.566+0000 7fee877fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee880fd280 0x7fee8810a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:16.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.566+0000 7fee877fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee880fd280 0x7fee8810a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45924/0 (socket says 172.21.15.23:45924) 2021-11-19T08:35:16.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.566+0000 7fee877fe700 1 -- 172.21.15.23:0/1604256836 learned_addr learned my addr 172.21.15.23:0/1604256836 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:16.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.566+0000 7fee877fe700 1 -- 172.21.15.23:0/1604256836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7800b200 con 0x7fee880fd280 2021-11-19T08:35:16.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.566+0000 7fee877fe700 1 --2- 172.21.15.23:0/1604256836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee880fd280 0x7fee8810a1c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fee78006c70 tx=0x7fee78016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:16.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.566+0000 7fee7ffff700 1 -- 172.21.15.23:0/1604256836 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee78011b40 con 0x7fee880fd280 2021-11-19T08:35:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.566+0000 7fee7ffff700 1 -- 172.21.15.23:0/1604256836 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee78011ca0 con 0x7fee880fd280 2021-11-19T08:35:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.566+0000 7fee8db75700 1 -- 172.21.15.23:0/1604256836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8810a8b0 con 0x7fee880fd280 2021-11-19T08:35:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.566+0000 7fee7ffff700 1 -- 172.21.15.23:0/1604256836 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee7801d430 con 0x7fee880fd280 2021-11-19T08:35:16.561 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.566+0000 7fee8db75700 1 -- 172.21.15.23:0/1604256836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8810acc0 con 0x7fee880fd280 2021-11-19T08:35:16.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.567+0000 7fee7ffff700 1 -- 172.21.15.23:0/1604256836 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fee7800f070 con 0x7fee880fd280 2021-11-19T08:35:16.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.567+0000 7fee7ffff700 1 -- 172.21.15.23:0/1604256836 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fee78027460 con 0x7fee880fd280 2021-11-19T08:35:16.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.568+0000 7fee8db75700 1 -- 172.21.15.23:0/1604256836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee88045ad0 con 0x7fee880fd280 2021-11-19T08:35:16.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.570+0000 7fee7ffff700 1 -- 172.21.15.23:0/1604256836 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fee78016070 con 0x7fee880fd280 2021-11-19T08:35:16.677 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.690+0000 7fee8db75700 1 -- 172.21.15.23:0/1604256836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fee880f9da0 con 0x7fee880fd280 2021-11-19T08:35:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.693+0000 7fee7ffff700 1 -- 172.21.15.23:0/1604256836 <== mon.0 v2:172.21.15.23:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fee7801d620 con 0x7fee880fd280 2021-11-19T08:35:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.693+0000 7fee7ffff700 1 -- 172.21.15.23:0/1604256836 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7fee780308f0 con 0x7fee880fd280 2021-11-19T08:35:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: [global] 2021-11-19T08:35:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: fsid = 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:35:16.686 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2021-11-19T08:35:16.686 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2021-11-19T08:35:16.686 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2021-11-19T08:35:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2021-11-19T08:35:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2021-11-19T08:35:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: [mgr] 2021-11-19T08:35:16.688 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2021-11-19T08:35:16.688 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.688 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: [osd] 2021-11-19T08:35:16.688 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2021-11-19T08:35:16.689 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2021-11-19T08:35:16.689 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2021-11-19T08:35:16.689 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2021-11-19T08:35:16.689 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2021-11-19T08:35:16.690 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2021-11-19T08:35:16.690 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.694+0000 7fee8db75700 1 -- 172.21.15.23:0/1604256836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee880fd280 msgr2=0x7fee8810a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:16.690 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.694+0000 7fee8db75700 1 --2- 172.21.15.23:0/1604256836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee880fd280 0x7fee8810a1c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fee78006c70 tx=0x7fee78016000).stop 2021-11-19T08:35:16.690 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.694+0000 7fee8db75700 1 -- 172.21.15.23:0/1604256836 shutdown_connections 2021-11-19T08:35:16.691 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.694+0000 7fee8db75700 1 --2- 172.21.15.23:0/1604256836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee880fd280 0x7fee8810a1c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:16.691 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.694+0000 7fee8db75700 1 -- 172.21.15.23:0/1604256836 >> 172.21.15.23:0/1604256836 conn(0x7fee880f5a80 msgr2=0x7fee880feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:16.691 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.695+0000 7fee8db75700 1 -- 172.21.15.23:0/1604256836 shutdown_connections 2021-11-19T08:35:16.691 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:16.695+0000 7fee8db75700 1 -- 172.21.15.23:0/1604256836 wait complete. 2021-11-19T08:35:17.102 INFO:teuthology.orchestra.run.smithi023.stderr:Generating new minimal ceph.conf... 2021-11-19T08:35:17.102 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z -v /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023:/var/lib/ceph/mon/ceph-smithi023:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi023/config 2021-11-19T08:35:17.241 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:16 smithi023 conmon[32798]: audit 2021-11-19T08:35:16.692270+0000 mon.smithi023 (mon.0) 8 : audit [INF] from='client.? 172.21.15.23:0/1604256836' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-11-19T08:35:17.241 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:16 smithi023 conmon[32798]: audit 2021-11-19T08:35:16.695746+0000 mon.smithi023 (mon.0) 9 : audit [INF] from='client.? 172.21.15.23:0/1604256836' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-11-19T08:35:17.553 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.561+0000 7fe27d29c700 1 Processor -- start 2021-11-19T08:35:17.553 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.561+0000 7fe27d29c700 1 -- start start 2021-11-19T08:35:17.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.561+0000 7fe27d29c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2780fa8f0 0x7fe2780fad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:17.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.561+0000 7fe27d29c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2780fb200 con 0x7fe2780fa8f0 2021-11-19T08:35:17.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.562+0000 7fe277fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2780fa8f0 0x7fe2780fad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:17.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.562+0000 7fe277fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2780fa8f0 0x7fe2780fad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45926/0 (socket says 172.21.15.23:45926) 2021-11-19T08:35:17.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.562+0000 7fe277fff700 1 -- 172.21.15.23:0/309875536 learned_addr learned my addr 172.21.15.23:0/309875536 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:17.555 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.562+0000 7fe277fff700 1 -- 172.21.15.23:0/309875536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2780fb340 con 0x7fe2780fa8f0 2021-11-19T08:35:17.555 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.562+0000 7fe277fff700 1 --2- 172.21.15.23:0/309875536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2780fa8f0 0x7fe2780fad10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fe26800b5f0 tx=0x7fe268006ee0).ready entity=mon.0 client_cookie=2d9f453e30797004 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:17.555 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.562+0000 7fe276ffd700 1 -- 172.21.15.23:0/309875536 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe26800d600 con 0x7fe2780fa8f0 2021-11-19T08:35:17.556 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.562+0000 7fe276ffd700 1 -- 172.21.15.23:0/309875536 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe26800dbd0 con 0x7fe2780fa8f0 2021-11-19T08:35:17.556 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.562+0000 7fe276ffd700 1 -- 172.21.15.23:0/309875536 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe268011c00 con 0x7fe2780fa8f0 2021-11-19T08:35:17.556 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.563+0000 7fe27d29c700 1 -- 172.21.15.23:0/309875536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2780fa8f0 msgr2=0x7fe2780fad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:17.556 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.563+0000 7fe27d29c700 1 --2- 172.21.15.23:0/309875536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2780fa8f0 0x7fe2780fad10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fe26800b5f0 tx=0x7fe268006ee0).stop 2021-11-19T08:35:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.563+0000 7fe27d29c700 1 -- 172.21.15.23:0/309875536 shutdown_connections 2021-11-19T08:35:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.563+0000 7fe27d29c700 1 --2- 172.21.15.23:0/309875536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2780fa8f0 0x7fe2780fad10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.563+0000 7fe27d29c700 1 -- 172.21.15.23:0/309875536 >> 172.21.15.23:0/309875536 conn(0x7fe2780f6130 msgr2=0x7fe2780f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.563+0000 7fe27d29c700 1 -- 172.21.15.23:0/309875536 shutdown_connections 2021-11-19T08:35:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.563+0000 7fe27d29c700 1 -- 172.21.15.23:0/309875536 wait complete. 2021-11-19T08:35:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.564+0000 7fe27d29c700 1 Processor -- start 2021-11-19T08:35:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.564+0000 7fe27d29c700 1 -- start start 2021-11-19T08:35:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.564+0000 7fe27d29c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2780fa8f0 0x7fe27810a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.564+0000 7fe27d29c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27810ac30 con 0x7fe2780fa8f0 2021-11-19T08:35:17.559 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.564+0000 7fe277fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2780fa8f0 0x7fe27810a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:17.559 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.564+0000 7fe277fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2780fa8f0 0x7fe27810a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45928/0 (socket says 172.21.15.23:45928) 2021-11-19T08:35:17.559 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.564+0000 7fe277fff700 1 -- 172.21.15.23:0/948354574 learned_addr learned my addr 172.21.15.23:0/948354574 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:17.559 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.565+0000 7fe277fff700 1 -- 172.21.15.23:0/948354574 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26800b200 con 0x7fe2780fa8f0 2021-11-19T08:35:17.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.565+0000 7fe277fff700 1 --2- 172.21.15.23:0/948354574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2780fa8f0 0x7fe27810a740 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe26800b1d0 tx=0x7fe268006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:17.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.565+0000 7fe274ff9700 1 -- 172.21.15.23:0/948354574 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe26801b410 con 0x7fe2780fa8f0 2021-11-19T08:35:17.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.565+0000 7fe274ff9700 1 -- 172.21.15.23:0/948354574 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe26801b9e0 con 0x7fe2780fa8f0 2021-11-19T08:35:17.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.565+0000 7fe27d29c700 1 -- 172.21.15.23:0/948354574 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe27810ae30 con 0x7fe2780fa8f0 2021-11-19T08:35:17.561 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.565+0000 7fe274ff9700 1 -- 172.21.15.23:0/948354574 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe26801a840 con 0x7fe2780fa8f0 2021-11-19T08:35:17.561 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.565+0000 7fe27d29c700 1 -- 172.21.15.23:0/948354574 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe27810b240 con 0x7fe2780fa8f0 2021-11-19T08:35:17.561 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.565+0000 7fe274ff9700 1 -- 172.21.15.23:0/948354574 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fe26800f070 con 0x7fe2780fa8f0 2021-11-19T08:35:17.561 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.565+0000 7fe274ff9700 1 -- 172.21.15.23:0/948354574 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe268025b80 con 0x7fe2780fa8f0 2021-11-19T08:35:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.566+0000 7fe27d29c700 1 -- 172.21.15.23:0/948354574 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe278045ad0 con 0x7fe2780fa8f0 2021-11-19T08:35:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.569+0000 7fe274ff9700 1 -- 172.21.15.23:0/948354574 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fe268016070 con 0x7fe2780fa8f0 2021-11-19T08:35:17.677 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.688+0000 7fe27d29c700 1 -- 172.21.15.23:0/948354574 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fe278104740 con 0x7fe2780fa8f0 2021-11-19T08:35:17.678 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.689+0000 7fe274ff9700 1 -- 172.21.15.23:0/948354574 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fe268025dc0 con 0x7fe2780fa8f0 2021-11-19T08:35:17.678 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.690+0000 7fe27d29c700 1 -- 172.21.15.23:0/948354574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2780fa8f0 msgr2=0x7fe27810a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:17.678 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.690+0000 7fe27d29c700 1 --2- 172.21.15.23:0/948354574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2780fa8f0 0x7fe27810a740 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe26800b1d0 tx=0x7fe268006ee0).stop 2021-11-19T08:35:17.678 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.690+0000 7fe27d29c700 1 -- 172.21.15.23:0/948354574 shutdown_connections 2021-11-19T08:35:17.679 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.690+0000 7fe27d29c700 1 --2- 172.21.15.23:0/948354574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2780fa8f0 0x7fe27810a740 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:17.679 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.690+0000 7fe27d29c700 1 -- 172.21.15.23:0/948354574 >> 172.21.15.23:0/948354574 conn(0x7fe2780f6130 msgr2=0x7fe2780f7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:17.679 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.690+0000 7fe27d29c700 1 -- 172.21.15.23:0/948354574 shutdown_connections 2021-11-19T08:35:17.679 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:17.690+0000 7fe27d29c700 1 -- 172.21.15.23:0/948354574 wait complete. 2021-11-19T08:35:18.039 INFO:teuthology.orchestra.run.smithi023.stderr:Restarting the monitor... 2021-11-19T08:35:18.039 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl restart ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi023 2021-11-19T08:35:18.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:17 smithi023 conmon[32798]: audit 2021-11-19T08:35:17.690878+0000 mon.smithi023 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.23:0/948354574' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:35:18.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:18 smithi023 systemd[1]: Stopping Ceph mon.smithi023 for 908b41ee-4913-11ec-8c2c-001a4aab830c... 2021-11-19T08:35:18.509 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:18 smithi023 conmon[32798]: debug 2021-11-19T08:35:18.173+0000 7f03bf55c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi023 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-19T08:35:18.510 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:18 smithi023 conmon[32798]: debug 2021-11-19T08:35:18.173+0000 7f03bf55c700 -1 mon.smithi023@0(leader) e1 *** Got Signal Terminated *** 2021-11-19T08:35:18.510 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:18 smithi023 conmon[32798]: 2021-11-19T08:35:18.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:18 smithi023 podman[33283]: ceph-908b41ee-4913-11ec-8c2c-001a4aab830c-mon.smithi023 2021-11-19T08:35:18.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:18 smithi023 systemd[1]: ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi023.service: Succeeded. 2021-11-19T08:35:18.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:18 smithi023 systemd[1]: Stopped Ceph mon.smithi023 for 908b41ee-4913-11ec-8c2c-001a4aab830c. 2021-11-19T08:35:18.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:18 smithi023 systemd[1]: Starting Ceph mon.smithi023 for 908b41ee-4913-11ec-8c2c-001a4aab830c... 2021-11-19T08:35:19.118 INFO:teuthology.orchestra.run.smithi023.stderr:Setting mon public_network to 172.21.0.0/20 2021-11-19T08:35:19.118 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2021-11-19T08:35:19.147 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:18 smithi023 podman[33450]: 2021-11-19T08:35:19.148 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.079+0000 7ff544a01700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-19T08:35:19.148 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.079+0000 7ff544a01700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2021-11-19T08:35:19.148 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.079+0000 7ff544a01700 0 pidfile_write: ignore empty --pid-file 2021-11-19T08:35:19.149 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.083+0000 7ff544a01700 0 load: jerasure load: lrc load: isa 2021-11-19T08:35:19.149 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: RocksDB version: 6.8.1 2021-11-19T08:35:19.149 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.149 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-19T08:35:19.149 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Compile date May 13 2021 2021-11-19T08:35:19.150 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: DB SUMMARY 2021-11-19T08:35:19.150 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.150 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: CURRENT file: CURRENT 2021-11-19T08:35:19.151 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.151 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: IDENTITY file: IDENTITY 2021-11-19T08:35:19.152 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.152 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2021-11-19T08:35:19.152 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.153 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi023/store.db dir, Total Num: 1, files: 000004.sst 2021-11-19T08:35:19.153 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.154 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi023/store.db: 000006.log size: 97733 ; 2021-11-19T08:35:19.154 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.154 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.error_if_exists: 0 2021-11-19T08:35:19.155 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.create_if_missing: 0 2021-11-19T08:35:19.155 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.paranoid_checks: 1 2021-11-19T08:35:19.156 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.env: 0x55deb409e1c0 2021-11-19T08:35:19.157 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.fs: Posix File System 2021-11-19T08:35:19.157 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.info_log: 0x55deb522f960 2021-11-19T08:35:19.158 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-19T08:35:19.159 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.statistics: (nil) 2021-11-19T08:35:19.159 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.159 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.use_fsync: 0 2021-11-19T08:35:19.160 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_log_file_size: 0 2021-11-19T08:35:19.160 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-19T08:35:19.161 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-19T08:35:19.162 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-19T08:35:19.162 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.162 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-19T08:35:19.163 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.allow_fallocate: 1 2021-11-19T08:35:19.163 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-19T08:35:19.163 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-19T08:35:19.163 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.use_direct_reads: 0 2021-11-19T08:35:19.164 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-19T08:35:19.164 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-19T08:35:19.164 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.db_log_dir: 2021-11-19T08:35:19.164 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi023/store.db 2021-11-19T08:35:19.164 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-19T08:35:19.165 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_subcompactions: 1 2021-11-19T08:35:19.165 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_background_flushes: -1 2021-11-19T08:35:19.165 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-19T08:35:19.165 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-19T08:35:19.166 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-19T08:35:19.166 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-19T08:35:19.166 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-19T08:35:19.166 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-19T08:35:19.167 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-19T08:35:19.167 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.write_buffer_manager: 0x55deb5ea4030 2021-11-19T08:35:19.167 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-19T08:35:19.167 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-19T08:35:19.167 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-19T08:35:19.168 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-19T08:35:19.170 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-19T08:35:19.170 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-19T08:35:19.170 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-19T08:35:19.170 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-19T08:35:19.171 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-19T08:35:19.171 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.unordered_write: 0 2021-11-19T08:35:19.171 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-19T08:35:19.171 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-19T08:35:19.171 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-19T08:35:19.172 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-19T08:35:19.172 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.row_cache: None 2021-11-19T08:35:19.172 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.wal_filter: None 2021-11-19T08:35:19.172 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.173 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-19T08:35:19.173 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-19T08:35:19.173 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.preserve_deletes: 0 2021-11-19T08:35:19.173 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.two_write_queues: 0 2021-11-19T08:35:19.174 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-19T08:35:19.174 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.atomic_flush: 0 2021-11-19T08:35:19.174 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-19T08:35:19.174 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-19T08:35:19.174 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-19T08:35:19.175 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.log_readahead_size: 0 2021-11-19T08:35:19.175 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-19T08:35:19.175 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_background_jobs: 2 2021-11-19T08:35:19.175 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_background_compactions: -1 2021-11-19T08:35:19.176 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-19T08:35:19.176 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-19T08:35:19.176 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-19T08:35:19.176 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-19T08:35:19.177 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-19T08:35:19.177 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-19T08:35:19.178 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-19T08:35:19.178 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-19T08:35:19.178 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_open_files: -1 2021-11-19T08:35:19.178 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-19T08:35:19.179 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.179 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-19T08:35:19.179 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-19T08:35:19.179 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-19T08:35:19.180 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Compression algorithms supported: 2021-11-19T08:35:19.180 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-19T08:35:19.180 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: kZSTD supported: 0 2021-11-19T08:35:19.180 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: kXpressCompression supported: 0 2021-11-19T08:35:19.181 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.181 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-19T08:35:19.181 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 bash[33374]: 04c7af95bd68b3622c60c0f27b5323fe0f46bdc3a3a4e023e397362ff3849839 2021-11-19T08:35:19.181 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: kLZ4Compression supported: 1 2021-11-19T08:35:19.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: kBZip2Compression supported: 0 2021-11-19T08:35:19.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: kZlibCompression supported: 1 2021-11-19T08:35:19.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: kSnappyCompression supported: 1 2021-11-19T08:35:19.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-19T08:35:19.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi023/store.db/MANIFEST-000005 2021-11-19T08:35:19.183 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.183 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-19T08:35:19.183 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.183 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-19T08:35:19.184 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.merge_operator: 2021-11-19T08:35:19.184 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compaction_filter: None 2021-11-19T08:35:19.184 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-19T08:35:19.184 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-19T08:35:19.185 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-19T08:35:19.185 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55deb5146128) 2021-11-19T08:35:19.185 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: cache_index_and_filter_blocks: 1 2021-11-19T08:35:19.185 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-19T08:35:19.186 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-19T08:35:19.186 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: pin_top_level_index_and_filter: 1 2021-11-19T08:35:19.186 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: index_type: 0 2021-11-19T08:35:19.186 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: data_block_index_type: 0 2021-11-19T08:35:19.188 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: index_shortening: 1 2021-11-19T08:35:19.188 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: data_block_hash_table_util_ratio: 0.750000 2021-11-19T08:35:19.188 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: hash_index_allow_collision: 1 2021-11-19T08:35:19.188 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: checksum: 1 2021-11-19T08:35:19.188 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: no_block_cache: 0 2021-11-19T08:35:19.189 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: block_cache: 0x55deb517cf10 2021-11-19T08:35:19.189 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: block_cache_name: BinnedLRUCache 2021-11-19T08:35:19.189 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: block_cache_options: 2021-11-19T08:35:19.189 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: capacity : 536870912 2021-11-19T08:35:19.190 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: num_shard_bits : 4 2021-11-19T08:35:19.190 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: strict_capacity_limit : 0 2021-11-19T08:35:19.190 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: high_pri_pool_ratio: 0.000 2021-11-19T08:35:19.190 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: block_cache_compressed: (nil) 2021-11-19T08:35:19.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: persistent_cache: (nil) 2021-11-19T08:35:19.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: block_size: 4096 2021-11-19T08:35:19.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: block_size_deviation: 10 2021-11-19T08:35:19.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: block_restart_interval: 16 2021-11-19T08:35:19.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: index_block_restart_interval: 1 2021-11-19T08:35:19.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: metadata_block_size: 4096 2021-11-19T08:35:19.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: partition_filters: 0 2021-11-19T08:35:19.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: use_delta_encoding: 1 2021-11-19T08:35:19.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-19T08:35:19.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: whole_key_filtering: 1 2021-11-19T08:35:19.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: verify_compression: 0 2021-11-19T08:35:19.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: read_amp_bytes_per_bit: 0 2021-11-19T08:35:19.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: format_version: 2 2021-11-19T08:35:19.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: enable_index_compression: 1 2021-11-19T08:35:19.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: block_align: 0 2021-11-19T08:35:19.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-19T08:35:19.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-19T08:35:19.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compression: NoCompression 2021-11-19T08:35:19.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-19T08:35:19.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-19T08:35:19.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-19T08:35:19.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.num_levels: 7 2021-11-19T08:35:19.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-19T08:35:19.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-19T08:35:19.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-19T08:35:19.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-19T08:35:19.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-19T08:35:19.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-19T08:35:19.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-19T08:35:19.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-19T08:35:19.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-19T08:35:19.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-19T08:35:19.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-19T08:35:19.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-19T08:35:19.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-19T08:35:19.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-19T08:35:19.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-19T08:35:19.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-19T08:35:19.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-19T08:35:19.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-19T08:35:19.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-19T08:35:19.203 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-19T08:35:19.203 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-19T08:35:19.203 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-19T08:35:19.203 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-19T08:35:19.204 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-19T08:35:19.204 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-19T08:35:19.204 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-19T08:35:19.204 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-19T08:35:19.204 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-19T08:35:19.205 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-19T08:35:19.205 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-19T08:35:19.205 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-19T08:35:19.205 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-19T08:35:19.206 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-19T08:35:19.206 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-19T08:35:19.206 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-19T08:35:19.207 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-19T08:35:19.207 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-19T08:35:19.207 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-19T08:35:19.207 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-19T08:35:19.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-19T08:35:19.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-19T08:35:19.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-19T08:35:19.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-19T08:35:19.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-19T08:35:19.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-19T08:35:19.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-19T08:35:19.210 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-19T08:35:19.210 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.table_properties_collectors: 2021-11-19T08:35:19.210 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.inplace_update_support: 0 2021-11-19T08:35:19.210 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-19T08:35:19.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-19T08:35:19.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-19T08:35:19.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-19T08:35:19.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.bloom_locality: 0 2021-11-19T08:35:19.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.max_successive_merges: 0 2021-11-19T08:35:19.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-19T08:35:19.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-19T08:35:19.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-19T08:35:19.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-19T08:35:19.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.ttl: 2592000 2021-11-19T08:35:19.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-19T08:35:19.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi023/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-19T08:35:19.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2021-11-19T08:35:19.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.084+0000 7ff544a01700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637310919086878, "job": 1, "event": "recovery_started", "log_files": [6]} 2021-11-19T08:35:19.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.085+0000 7ff544a01700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2021-11-19T08:35:19.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.085+0000 7ff544a01700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-11-19T08:35:19.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.085+0000 7ff544a01700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637310919087765, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 93403, "table_properties": {"data_size": 91503, "index_size": 377, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12317, "raw_average_key_size": 49, "raw_value_size": 84659, "raw_average_value_size": 338, "num_data_blocks": 12, "num_entries": 250, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637310919, "oldest_key_time": 3, "file_creation_time": 0}} 2021-11-19T08:35:19.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.085+0000 7ff544a01700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2021-11-19T08:35:19.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.091+0000 7ff544a01700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637310919092956, "job": 1, "event": "recovery_finished"} 2021-11-19T08:35:19.216 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.091+0000 7ff544a01700 4 rocksdb: DB pointer 0x55deb5241800 2021-11-19T08:35:19.216 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.092+0000 7ff52b819700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-19T08:35:19.216 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.092+0000 7ff52b819700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-19T08:35:19.216 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: ** DB Stats ** 2021-11-19T08:35:19.217 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-19T08:35:19.217 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-19T08:35:19.217 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-19T08:35:19.217 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-19T08:35:19.217 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-19T08:35:19.218 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-19T08:35:19.218 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-19T08:35:19.218 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.218 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: ** Compaction Stats [default] ** 2021-11-19T08:35:19.218 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-19T08:35:19.219 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-19T08:35:19.219 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: L0 2/0 92.91 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.0 0.00 0.00 1 0.001 0 0 2021-11-19T08:35:19.219 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Sum 2/0 92.91 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.0 0.00 0.00 1 0.001 0 0 2021-11-19T08:35:19.219 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.0 0.00 0.00 1 0.001 0 0 2021-11-19T08:35:19.220 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.220 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: ** Compaction Stats [default] ** 2021-11-19T08:35:19.220 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-19T08:35:19.220 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-19T08:35:19.220 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 162.0 0.00 0.00 1 0.001 0 0 2021-11-19T08:35:19.221 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-19T08:35:19.221 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-19T08:35:19.221 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-19T08:35:19.221 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: AddFile(Total Files): cumulative 0, interval 0 2021-11-19T08:35:19.222 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-19T08:35:19.222 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: AddFile(Keys): cumulative 0, interval 0 2021-11-19T08:35:19.222 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Cumulative compaction: 0.00 GB write, 11.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-19T08:35:19.222 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Interval compaction: 0.00 GB write, 11.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-19T08:35:19.222 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-19T08:35:19.223 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.223 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: ** File Read Latency Histogram By Level [default] ** 2021-11-19T08:35:19.223 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.223 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: ** Compaction Stats [default] ** 2021-11-19T08:35:19.224 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-19T08:35:19.224 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-19T08:35:19.224 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: L0 2/0 92.91 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.0 0.00 0.00 1 0.001 0 0 2021-11-19T08:35:19.224 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Sum 2/0 92.91 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.0 0.00 0.00 1 0.001 0 0 2021-11-19T08:35:19.224 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-19T08:35:19.225 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.225 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: ** Compaction Stats [default] ** 2021-11-19T08:35:19.225 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-19T08:35:19.225 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-19T08:35:19.226 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 162.0 0.00 0.00 1 0.001 0 0 2021-11-19T08:35:19.226 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-19T08:35:19.226 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-19T08:35:19.226 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-19T08:35:19.226 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: AddFile(Total Files): cumulative 0, interval 0 2021-11-19T08:35:19.227 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-19T08:35:19.227 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: AddFile(Keys): cumulative 0, interval 0 2021-11-19T08:35:19.227 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Cumulative compaction: 0.00 GB write, 11.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-19T08:35:19.227 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-19T08:35:19.228 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-19T08:35:19.228 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.228 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: ** File Read Latency Histogram By Level [default] ** 2021-11-19T08:35:19.228 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.228 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.092+0000 7ff544a01700 0 starting mon.smithi023 rank 0 at public addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] at bind addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi023 fsid 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:35:19.229 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.093+0000 7ff544a01700 1 mon.smithi023@-1(???) e1 preinit fsid 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:35:19.229 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.093+0000 7ff544a01700 0 mon.smithi023@-1(???).mds e1 new map 2021-11-19T08:35:19.229 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.093+0000 7ff544a01700 0 mon.smithi023@-1(???).mds e1 print_map 2021-11-19T08:35:19.229 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: e1 2021-11-19T08:35:19.230 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: enable_multiple, ever_enabled_multiple: 1,1 2021-11-19T08:35:19.230 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-19T08:35:19.230 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: legacy client fscid: -1 2021-11-19T08:35:19.230 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.230 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: No filesystems configured 2021-11-19T08:35:19.231 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35:19.231 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.093+0000 7ff544a01700 0 mon.smithi023@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2021-11-19T08:35:19.231 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.093+0000 7ff544a01700 0 mon.smithi023@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-19T08:35:19.231 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.093+0000 7ff544a01700 0 mon.smithi023@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-19T08:35:19.232 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.093+0000 7ff544a01700 0 mon.smithi023@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-11-19T08:35:19.232 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: audit 2021-11-19T08:35:19.232 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 2021-11-19T08:35: 2021-11-19T08:35:19.232 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: 17.690878+0000 mon.smithi023 (mon.0 2021-11-19T08:35:19.232 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: ) 10 : audit [DBG] from='client.? 172.21.15.23:0/948354574' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:35:19.233 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: debug 2021-11-19T08:35:19.093+0000 7ff544a01700 1 mon.smithi023@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2021-11-19T08:35:19.233 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 systemd[1]: Started Ceph mon.smithi023 for 908b41ee-4913-11ec-8c2c-001a4aab830c. 2021-11-19T08:35:19.233 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: cluster 2021-11-19T08:35:19.110646+0000 2021-11-19T08:35:19.233 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: mon.smithi023 (mon.0) 1 : cluster [INF] mon.smithi023 is new leader, mons smithi023 in quorum (ranks 0) 2021-11-19T08:35:19.233 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: cluster 2021-11-19T08:35:19.110687+0000 mon.smithi023 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi023=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} 2021-11-19T08:35:19.234 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: cluster 2021-11-19T08:35:19.110716+0000 mon.smithi023 (mon.0) 3 : cluster [DBG] fsmap 2021-11-19T08:35:19.234 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: cluster 2021-11-19 2021-11-19T08:35:19.234 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: T08:35:19.110741+0000 mon.smithi023 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-11-19T08:35:19.234 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: cluster 2021-11-19T08:35 2021-11-19T08:35:19.234 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: :19.111115+0000 mon.smithi023 (mon.0 2021-11-19T08:35:19.235 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:19 smithi023 conmon[33490]: ) 5 : cluster [DBG] mgrmap e1: no daemons active 2021-11-19T08:35:19.611 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.620+0000 7f06ff7e6700 1 Processor -- start 2021-11-19T08:35:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.620+0000 7f06ff7e6700 1 -- start start 2021-11-19T08:35:19.612 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.620+0000 7f06ff7e6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f806fbf0 0x7f06f80fe870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.620+0000 7f06ff7e6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80fed60 con 0x7f06f806fbf0 2021-11-19T08:35:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.621+0000 7f06fd582700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f806fbf0 0x7f06f80fe870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:19.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.621+0000 7f06fd582700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f806fbf0 0x7f06f80fe870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45930/0 (socket says 172.21.15.23:45930) 2021-11-19T08:35:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.621+0000 7f06fd582700 1 -- 172.21.15.23:0/186621795 learned_addr learned my addr 172.21.15.23:0/186621795 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.621+0000 7f06fd582700 1 -- 172.21.15.23:0/186621795 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80feea0 con 0x7f06f806fbf0 2021-11-19T08:35:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.621+0000 7f06fd582700 1 --2- 172.21.15.23:0/186621795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f806fbf0 0x7f06f80fe870 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f06e8009960 tx=0x7f06e80092d0).ready entity=mon.0 client_cookie=d23a81deb561cdf2 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:19.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.621+0000 7f06f7fff700 1 -- 172.21.15.23:0/186621795 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06e800ba50 con 0x7f06f806fbf0 2021-11-19T08:35:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.621+0000 7f06f7fff700 1 -- 172.21.15.23:0/186621795 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f06e800bbb0 con 0x7f06f806fbf0 2021-11-19T08:35:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.621+0000 7f06f7fff700 1 -- 172.21.15.23:0/186621795 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06e80103a0 con 0x7f06f806fbf0 2021-11-19T08:35:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.622+0000 7f06ff7e6700 1 -- 172.21.15.23:0/186621795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f806fbf0 msgr2=0x7f06f80fe870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.622+0000 7f06ff7e6700 1 --2- 172.21.15.23:0/186621795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f806fbf0 0x7f06f80fe870 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f06e8009960 tx=0x7f06e80092d0).stop 2021-11-19T08:35:19.615 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.622+0000 7f06ff7e6700 1 -- 172.21.15.23:0/186621795 shutdown_connections 2021-11-19T08:35:19.616 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.622+0000 7f06ff7e6700 1 --2- 172.21.15.23:0/186621795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f806fbf0 0x7f06f80fe870 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:19.616 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.622+0000 7f06ff7e6700 1 -- 172.21.15.23:0/186621795 >> 172.21.15.23:0/186621795 conn(0x7f06f80f6310 msgr2=0x7f06f80f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:19.616 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.622+0000 7f06ff7e6700 1 -- 172.21.15.23:0/186621795 shutdown_connections 2021-11-19T08:35:19.616 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.622+0000 7f06ff7e6700 1 -- 172.21.15.23:0/186621795 wait complete. 2021-11-19T08:35:19.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.623+0000 7f06ff7e6700 1 Processor -- start 2021-11-19T08:35:19.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.623+0000 7f06ff7e6700 1 -- start start 2021-11-19T08:35:19.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.623+0000 7f06ff7e6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f806fbf0 0x7f06f80fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:19.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.623+0000 7f06ff7e6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f80fa770 con 0x7f06f806fbf0 2021-11-19T08:35:19.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.623+0000 7f06fd582700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f806fbf0 0x7f06f80fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.623+0000 7f06fd582700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f806fbf0 0x7f06f80fc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45932/0 (socket says 172.21.15.23:45932) 2021-11-19T08:35:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.623+0000 7f06fd582700 1 -- 172.21.15.23:0/2588814332 learned_addr learned my addr 172.21.15.23:0/2588814332 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.624+0000 7f06fd582700 1 -- 172.21.15.23:0/2588814332 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e8009570 con 0x7f06f806fbf0 2021-11-19T08:35:19.618 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.624+0000 7f06fd582700 1 --2- 172.21.15.23:0/2588814332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f806fbf0 0x7f06f80fc2f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f06e800bf30 tx=0x7f06e80092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:19.619 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.624+0000 7f06f5ffb700 1 -- 172.21.15.23:0/2588814332 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06e80103a0 con 0x7f06f806fbf0 2021-11-19T08:35:19.619 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.624+0000 7f06f5ffb700 1 -- 172.21.15.23:0/2588814332 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f06e8010c60 con 0x7f06f806fbf0 2021-11-19T08:35:19.619 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.624+0000 7f06f5ffb700 1 -- 172.21.15.23:0/2588814332 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06e801bc20 con 0x7f06f806fbf0 2021-11-19T08:35:19.620 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.624+0000 7f06ff7e6700 1 -- 172.21.15.23:0/2588814332 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f80fa910 con 0x7f06f806fbf0 2021-11-19T08:35:19.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.624+0000 7f06ff7e6700 1 -- 172.21.15.23:0/2588814332 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f80fad20 con 0x7f06f806fbf0 2021-11-19T08:35:19.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.624+0000 7f06f5ffb700 1 -- 172.21.15.23:0/2588814332 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f06e802b070 con 0x7f06f806fbf0 2021-11-19T08:35:19.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.624+0000 7f06f5ffb700 1 -- 172.21.15.23:0/2588814332 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f06e801a710 con 0x7f06f806fbf0 2021-11-19T08:35:19.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.625+0000 7f06ff7e6700 1 -- 172.21.15.23:0/2588814332 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f8045ad0 con 0x7f06f806fbf0 2021-11-19T08:35:19.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.628+0000 7f06f5ffb700 1 -- 172.21.15.23:0/2588814332 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f06e801f020 con 0x7f06f806fbf0 2021-11-19T08:35:19.734 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.747+0000 7f06ff7e6700 1 -- 172.21.15.23:0/2588814332 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f06f80008d0 con 0x7f06f806fbf0 2021-11-19T08:35:19.736 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.749+0000 7f06f5ffb700 1 -- 172.21.15.23:0/2588814332 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f06e8009dd0 con 0x7f06f806fbf0 2021-11-19T08:35:19.738 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.750+0000 7f06ff7e6700 1 -- 172.21.15.23:0/2588814332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f806fbf0 msgr2=0x7f06f80fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:19.739 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.750+0000 7f06ff7e6700 1 --2- 172.21.15.23:0/2588814332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f806fbf0 0x7f06f80fc2f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f06e800bf30 tx=0x7f06e80092d0).stop 2021-11-19T08:35:19.739 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.750+0000 7f06ff7e6700 1 -- 172.21.15.23:0/2588814332 shutdown_connections 2021-11-19T08:35:19.740 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.750+0000 7f06ff7e6700 1 --2- 172.21.15.23:0/2588814332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f806fbf0 0x7f06f80fc2f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:19.740 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.750+0000 7f06ff7e6700 1 -- 172.21.15.23:0/2588814332 >> 172.21.15.23:0/2588814332 conn(0x7f06f80f6310 msgr2=0x7f06f80f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:19.740 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.750+0000 7f06ff7e6700 1 -- 172.21.15.23:0/2588814332 shutdown_connections 2021-11-19T08:35:19.741 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:19.750+0000 7f06ff7e6700 1 -- 172.21.15.23:0/2588814332 wait complete. 2021-11-19T08:35:20.260 INFO:teuthology.orchestra.run.smithi023.stderr:Wrote config to /etc/ceph/ceph.conf 2021-11-19T08:35:20.261 INFO:teuthology.orchestra.run.smithi023.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2021-11-19T08:35:20.261 INFO:teuthology.orchestra.run.smithi023.stderr:Creating mgr... 2021-11-19T08:35:20.261 INFO:teuthology.orchestra.run.smithi023.stderr:Verifying port 9283 ... 2021-11-19T08:35:20.262 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl daemon-reload 2021-11-19T08:35:20.357 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl stop ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mgr.smithi023.ehlpcl 2021-11-19T08:35:20.383 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl reset-failed ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mgr.smithi023.ehlpcl 2021-11-19T08:35:20.389 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Failed to reset failed state of unit ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mgr.smithi023.ehlpcl.service: Unit ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mgr.smithi023.ehlpcl.service not loaded. 2021-11-19T08:35:20.389 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl enable ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mgr.smithi023.ehlpcl 2021-11-19T08:35:20.396 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Created symlink /etc/systemd/system/ceph-908b41ee-4913-11ec-8c2c-001a4aab830c.target.wants/ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mgr.smithi023.ehlpcl.service → /etc/systemd/system/ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@.service. 2021-11-19T08:35:20.477 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl start ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mgr.smithi023.ehlpcl 2021-11-19T08:35:20.999 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:20 smithi023 conmon[33490]: audit 2021-11-19T08:35:19.751396+0000 mon.smithi023 (mon.0) 6 : audit [INF] from='client.? 172.21.15.23:0/2588814332' entity='client.admin' 2021-11-19T08:35:21.007 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-19T08:35:21.016 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: disabled 2021-11-19T08:35:21.016 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active firewalld.service 2021-11-19T08:35:21.023 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: inactive 2021-11-19T08:35:21.024 INFO:teuthology.orchestra.run.smithi023.stderr:firewalld.service is not enabled 2021-11-19T08:35:21.024 INFO:teuthology.orchestra.run.smithi023.stderr:Not possible to enable service . firewalld.service is not available 2021-11-19T08:35:21.025 INFO:teuthology.orchestra.run.smithi023.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-11-19T08:35:21.025 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-19T08:35:21.032 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: disabled 2021-11-19T08:35:21.032 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active firewalld.service 2021-11-19T08:35:21.039 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: inactive 2021-11-19T08:35:21.040 INFO:teuthology.orchestra.run.smithi023.stderr:firewalld.service is not enabled 2021-11-19T08:35:21.040 INFO:teuthology.orchestra.run.smithi023.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2021-11-19T08:35:21.041 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for mgr to start... 2021-11-19T08:35:21.041 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for mgr... 2021-11-19T08:35:21.041 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-19T08:35:21.518 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.520+0000 7f6aef59e700 1 Processor -- start 2021-11-19T08:35:21.519 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.520+0000 7f6aef59e700 1 -- start start 2021-11-19T08:35:21.519 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.520+0000 7f6aef59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af006fb70 0x7f6af00fe710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:21.520 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.520+0000 7f6aef59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af006ff90 con 0x7f6af006fb70 2021-11-19T08:35:21.520 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.521+0000 7f6aee59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af006fb70 0x7f6af00fe710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:21.521 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.521+0000 7f6aee59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af006fb70 0x7f6af00fe710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45938/0 (socket says 172.21.15.23:45938) 2021-11-19T08:35:21.521 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.521+0000 7f6aee59c700 1 -- 172.21.15.23:0/2033832571 learned_addr learned my addr 172.21.15.23:0/2033832571 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:21.522 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.521+0000 7f6aee59c700 1 -- 172.21.15.23:0/2033832571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af00fec00 con 0x7f6af006fb70 2021-11-19T08:35:21.522 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.521+0000 7f6aee59c700 1 --2- 172.21.15.23:0/2033832571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af006fb70 0x7f6af00fe710 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f6ae000b5f0 tx=0x7f6ae0006ee0).ready entity=mon.0 client_cookie=e4002c7efed595b8 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:21.523 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.522+0000 7f6aed59a700 1 -- 172.21.15.23:0/2033832571 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ae000d600 con 0x7f6af006fb70 2021-11-19T08:35:21.523 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.522+0000 7f6aed59a700 1 -- 172.21.15.23:0/2033832571 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6ae000dbd0 con 0x7f6af006fb70 2021-11-19T08:35:21.524 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.522+0000 7f6aed59a700 1 -- 172.21.15.23:0/2033832571 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ae0011c00 con 0x7f6af006fb70 2021-11-19T08:35:21.524 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.522+0000 7f6aef59e700 1 -- 172.21.15.23:0/2033832571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af006fb70 msgr2=0x7f6af00fe710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:21.525 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.522+0000 7f6aef59e700 1 --2- 172.21.15.23:0/2033832571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af006fb70 0x7f6af00fe710 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f6ae000b5f0 tx=0x7f6ae0006ee0).stop 2021-11-19T08:35:21.525 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.522+0000 7f6aef59e700 1 -- 172.21.15.23:0/2033832571 shutdown_connections 2021-11-19T08:35:21.526 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.522+0000 7f6aef59e700 1 --2- 172.21.15.23:0/2033832571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af006fb70 0x7f6af00fe710 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:21.526 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.522+0000 7f6aef59e700 1 -- 172.21.15.23:0/2033832571 >> 172.21.15.23:0/2033832571 conn(0x7f6af00f6130 msgr2=0x7f6af00f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:21.526 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.523+0000 7f6aef59e700 1 -- 172.21.15.23:0/2033832571 shutdown_connections 2021-11-19T08:35:21.527 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.523+0000 7f6aef59e700 1 -- 172.21.15.23:0/2033832571 wait complete. 2021-11-19T08:35:21.527 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.523+0000 7f6aef59e700 1 Processor -- start 2021-11-19T08:35:21.528 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.523+0000 7f6aef59e700 1 -- start start 2021-11-19T08:35:21.529 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.523+0000 7f6aef59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af006fb70 0x7f6af010a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:21.529 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.523+0000 7f6aef59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af010ac70 con 0x7f6af006fb70 2021-11-19T08:35:21.530 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.524+0000 7f6aee59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af006fb70 0x7f6af010a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:21.530 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.524+0000 7f6aee59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af006fb70 0x7f6af010a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45940/0 (socket says 172.21.15.23:45940) 2021-11-19T08:35:21.530 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.524+0000 7f6aee59c700 1 -- 172.21.15.23:0/3318194187 learned_addr learned my addr 172.21.15.23:0/3318194187 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:21.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.524+0000 7f6aee59c700 1 -- 172.21.15.23:0/3318194187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae000b200 con 0x7f6af006fb70 2021-11-19T08:35:21.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.524+0000 7f6aee59c700 1 --2- 172.21.15.23:0/3318194187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af006fb70 0x7f6af010a780 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f6ae000b190 tx=0x7f6ae0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:21.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.524+0000 7f6adeffd700 1 -- 172.21.15.23:0/3318194187 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ae001b410 con 0x7f6af006fb70 2021-11-19T08:35:21.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.524+0000 7f6adeffd700 1 -- 172.21.15.23:0/3318194187 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6ae001b9e0 con 0x7f6af006fb70 2021-11-19T08:35:21.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.524+0000 7f6aef59e700 1 -- 172.21.15.23:0/3318194187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af010ae70 con 0x7f6af006fb70 2021-11-19T08:35:21.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.524+0000 7f6aef59e700 1 -- 172.21.15.23:0/3318194187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af010b280 con 0x7f6af006fb70 2021-11-19T08:35:21.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.525+0000 7f6adeffd700 1 -- 172.21.15.23:0/3318194187 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ae001a880 con 0x7f6af006fb70 2021-11-19T08:35:21.534 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.525+0000 7f6adeffd700 1 -- 172.21.15.23:0/3318194187 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f6ae000f070 con 0x7f6af006fb70 2021-11-19T08:35:21.534 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.525+0000 7f6adeffd700 1 -- 172.21.15.23:0/3318194187 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6ae0025c20 con 0x7f6af006fb70 2021-11-19T08:35:21.535 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.526+0000 7f6aef59e700 1 -- 172.21.15.23:0/3318194187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6acc004fa0 con 0x7f6af006fb70 2021-11-19T08:35:21.535 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.528+0000 7f6adeffd700 1 -- 172.21.15.23:0/3318194187 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f6ae0016070 con 0x7f6af006fb70 2021-11-19T08:35:21.642 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.654+0000 7f6aef59e700 1 -- 172.21.15.23:0/3318194187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6acc005670 con 0x7f6af006fb70 2021-11-19T08:35:21.651 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.655+0000 7f6adeffd700 1 -- 172.21.15.23:0/3318194187 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f6ae0006a70 con 0x7f6af006fb70 2021-11-19T08:35:21.652 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.652 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2021-11-19T08:35:21.652 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsid": "908b41ee-4913-11ec-8c2c-001a4aab830c", 2021-11-19T08:35:21.653 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "health": { 2021-11-19T08:35:21.653 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-19T08:35:21.653 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "checks": {}, 2021-11-19T08:35:21.654 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mutes": [] 2021-11-19T08:35:21.654 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:21.655 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-19T08:35:21.655 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum": [ 2021-11-19T08:35:21.655 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 0 2021-11-19T08:35:21.656 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2021-11-19T08:35:21.656 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-19T08:35:21.656 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "smithi023" 2021-11-19T08:35:21.656 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2021-11-19T08:35:21.657 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_age": 2, 2021-11-19T08:35:21.657 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "monmap": { 2021-11-19T08:35:21.657 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:21.658 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-19T08:35:21.658 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-19T08:35:21.658 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:21.658 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osdmap": { 2021-11-19T08:35:21.659 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:21.659 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-19T08:35:21.659 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-19T08:35:21.660 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-19T08:35:21.660 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-19T08:35:21.660 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-19T08:35:21.660 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-19T08:35:21.661 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:21.661 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgmap": { 2021-11-19T08:35:21.661 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-19T08:35:21.661 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-19T08:35:21.662 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-19T08:35:21.662 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-19T08:35:21.662 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-19T08:35:21.662 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-19T08:35:21.663 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-19T08:35:21.663 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-19T08:35:21.663 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:21.663 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsmap": { 2021-11-19T08:35:21.664 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:21.664 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-19T08:35:21.665 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-19T08:35:21.665 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:21.665 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-19T08:35:21.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "available": false, 2021-11-19T08:35:21.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-19T08:35:21.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modules": [ 2021-11-19T08:35:21.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "iostat", 2021-11-19T08:35:21.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "restful" 2021-11-19T08:35:21.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2021-11-19T08:35:21.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2021-11-19T08:35:21.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:21.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "servicemap": { 2021-11-19T08:35:21.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:21.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modified": "2021-11-19T08:35:14.953374+0000", 2021-11-19T08:35:21.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2021-11-19T08:35:21.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:21.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-19T08:35:21.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2021-11-19T08:35:21.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.656+0000 7f6aef59e700 1 -- 172.21.15.23:0/3318194187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af006fb70 msgr2=0x7f6af010a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:21.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.656+0000 7f6aef59e700 1 --2- 172.21.15.23:0/3318194187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af006fb70 0x7f6af010a780 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f6ae000b190 tx=0x7f6ae0006ee0).stop 2021-11-19T08:35:21.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.656+0000 7f6aef59e700 1 -- 172.21.15.23:0/3318194187 shutdown_connections 2021-11-19T08:35:21.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.656+0000 7f6aef59e700 1 --2- 172.21.15.23:0/3318194187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af006fb70 0x7f6af010a780 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:21.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.656+0000 7f6aef59e700 1 -- 172.21.15.23:0/3318194187 >> 172.21.15.23:0/3318194187 conn(0x7f6af00f6130 msgr2=0x7f6af00f7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:21.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.657+0000 7f6aef59e700 1 -- 172.21.15.23:0/3318194187 shutdown_connections 2021-11-19T08:35:21.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:21.657+0000 7f6aef59e700 1 -- 172.21.15.23:0/3318194187 wait complete. 2021-11-19T08:35:21.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:21 smithi023 conmon[33490]: audit 2021-11-19T08:35:21.657013+0000 mon.smithi023 ( 2021-11-19T08:35:21.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:21 smithi023 conmon[33490]: mon.0) 7 : audit [DBG] from='client.? 172.21.15.23:0/3318194187' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-19T08:35:21.979 INFO:teuthology.orchestra.run.smithi023.stderr:mgr not available, waiting (1/15)... 2021-11-19T08:35:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-19T08:35:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.531+0000 7f29cb83b700 1 Processor -- start 2021-11-19T08:35:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.531+0000 7f29cb83b700 1 -- start start 2021-11-19T08:35:24.524 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.532+0000 7f29cb83b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c40faae0 0x7f29c40faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:24.524 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.532+0000 7f29cb83b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40fb480 con 0x7f29c40faae0 2021-11-19T08:35:24.524 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.532+0000 7f29c95d7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c40faae0 0x7f29c40faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:24.524 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.532+0000 7f29c95d7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c40faae0 0x7f29c40faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45942/0 (socket says 172.21.15.23:45942) 2021-11-19T08:35:24.524 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.532+0000 7f29c95d7700 1 -- 172.21.15.23:0/4155486902 learned_addr learned my addr 172.21.15.23:0/4155486902 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:24.525 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.532+0000 7f29c95d7700 1 -- 172.21.15.23:0/4155486902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c40fb5c0 con 0x7f29c40faae0 2021-11-19T08:35:24.525 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.532+0000 7f29c95d7700 1 --2- 172.21.15.23:0/4155486902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c40faae0 0x7f29c40faf00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f29c000b5f0 tx=0x7f29c0006ee0).ready entity=mon.0 client_cookie=76a6ea2ab6157362 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:24.525 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.532+0000 7f29b7fff700 1 -- 172.21.15.23:0/4155486902 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29c000d600 con 0x7f29c40faae0 2021-11-19T08:35:24.526 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.533+0000 7f29b7fff700 1 -- 172.21.15.23:0/4155486902 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f29c000dbd0 con 0x7f29c40faae0 2021-11-19T08:35:24.526 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.533+0000 7f29b7fff700 1 -- 172.21.15.23:0/4155486902 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29c0011c00 con 0x7f29c40faae0 2021-11-19T08:35:24.526 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.533+0000 7f29cb83b700 1 -- 172.21.15.23:0/4155486902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c40faae0 msgr2=0x7f29c40faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:24.526 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.533+0000 7f29cb83b700 1 --2- 172.21.15.23:0/4155486902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c40faae0 0x7f29c40faf00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f29c000b5f0 tx=0x7f29c0006ee0).stop 2021-11-19T08:35:24.527 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.533+0000 7f29cb83b700 1 -- 172.21.15.23:0/4155486902 shutdown_connections 2021-11-19T08:35:24.527 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.533+0000 7f29cb83b700 1 --2- 172.21.15.23:0/4155486902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c40faae0 0x7f29c40faf00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.533+0000 7f29cb83b700 1 -- 172.21.15.23:0/4155486902 >> 172.21.15.23:0/4155486902 conn(0x7f29c40f6280 msgr2=0x7f29c40f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.534+0000 7f29cb83b700 1 -- 172.21.15.23:0/4155486902 shutdown_connections 2021-11-19T08:35:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.534+0000 7f29cb83b700 1 -- 172.21.15.23:0/4155486902 wait complete. 2021-11-19T08:35:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.534+0000 7f29cb83b700 1 Processor -- start 2021-11-19T08:35:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.534+0000 7f29cb83b700 1 -- start start 2021-11-19T08:35:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.535+0000 7f29cb83b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c40faae0 0x7f29c410cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.535+0000 7f29cb83b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c410d0c0 con 0x7f29c40faae0 2021-11-19T08:35:24.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.535+0000 7f29c95d7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c40faae0 0x7f29c410cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:24.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.535+0000 7f29c95d7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c40faae0 0x7f29c410cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45944/0 (socket says 172.21.15.23:45944) 2021-11-19T08:35:24.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.535+0000 7f29c95d7700 1 -- 172.21.15.23:0/801581159 learned_addr learned my addr 172.21.15.23:0/801581159 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:24.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.535+0000 7f29c95d7700 1 -- 172.21.15.23:0/801581159 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c000b200 con 0x7f29c40faae0 2021-11-19T08:35:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.535+0000 7f29c95d7700 1 --2- 172.21.15.23:0/801581159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c40faae0 0x7f29c410cbd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f29c000b1d0 tx=0x7f29c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.535+0000 7f29b5ffb700 1 -- 172.21.15.23:0/801581159 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29c0011a20 con 0x7f29c40faae0 2021-11-19T08:35:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.535+0000 7f29b5ffb700 1 -- 172.21.15.23:0/801581159 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f29c001b880 con 0x7f29c40faae0 2021-11-19T08:35:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.535+0000 7f29b5ffb700 1 -- 172.21.15.23:0/801581159 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29c001a700 con 0x7f29c40faae0 2021-11-19T08:35:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.535+0000 7f29cb83b700 1 -- 172.21.15.23:0/801581159 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c410d2c0 con 0x7f29c40faae0 2021-11-19T08:35:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.536+0000 7f29cb83b700 1 -- 172.21.15.23:0/801581159 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c410d6d0 con 0x7f29c40faae0 2021-11-19T08:35:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.537+0000 7f29b5ffb700 1 -- 172.21.15.23:0/801581159 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f29c000f070 con 0x7f29c40faae0 2021-11-19T08:35:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.537+0000 7f29b5ffb700 1 -- 172.21.15.23:0/801581159 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f29c0025ac0 con 0x7f29c40faae0 2021-11-19T08:35:24.535 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.537+0000 7f29cb83b700 1 -- 172.21.15.23:0/801581159 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c4045ad0 con 0x7f29c40faae0 2021-11-19T08:35:24.535 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.539+0000 7f29b5ffb700 1 -- 172.21.15.23:0/801581159 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f29c0016070 con 0x7f29c40faae0 2021-11-19T08:35:24.660 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.665+0000 7f29cb83b700 1 -- 172.21.15.23:0/801581159 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f29c4106d10 con 0x7f29c40faae0 2021-11-19T08:35:24.661 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.666+0000 7f29b5ffb700 1 -- 172.21.15.23:0/801581159 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f29c001adb0 con 0x7f29c40faae0 2021-11-19T08:35:24.661 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.661 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2021-11-19T08:35:24.661 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsid": "908b41ee-4913-11ec-8c2c-001a4aab830c", 2021-11-19T08:35:24.662 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "health": { 2021-11-19T08:35:24.662 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-19T08:35:24.662 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "checks": {}, 2021-11-19T08:35:24.662 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mutes": [] 2021-11-19T08:35:24.663 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:24.663 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-19T08:35:24.663 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum": [ 2021-11-19T08:35:24.663 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 0 2021-11-19T08:35:24.664 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2021-11-19T08:35:24.664 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-19T08:35:24.664 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "smithi023" 2021-11-19T08:35:24.664 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2021-11-19T08:35:24.665 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_age": 5, 2021-11-19T08:35:24.665 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "monmap": { 2021-11-19T08:35:24.665 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:24.665 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-19T08:35:24.665 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-19T08:35:24.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:24.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osdmap": { 2021-11-19T08:35:24.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:24.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-19T08:35:24.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-19T08:35:24.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-19T08:35:24.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-19T08:35:24.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-19T08:35:24.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-19T08:35:24.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:24.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgmap": { 2021-11-19T08:35:24.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-19T08:35:24.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-19T08:35:24.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-19T08:35:24.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-19T08:35:24.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-19T08:35:24.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-19T08:35:24.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-19T08:35:24.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-19T08:35:24.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:24.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsmap": { 2021-11-19T08:35:24.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:24.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-19T08:35:24.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-19T08:35:24.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:24.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-19T08:35:24.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "available": false, 2021-11-19T08:35:24.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-19T08:35:24.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modules": [ 2021-11-19T08:35:24.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "iostat", 2021-11-19T08:35:24.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "restful" 2021-11-19T08:35:24.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2021-11-19T08:35:24.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2021-11-19T08:35:24.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:24.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "servicemap": { 2021-11-19T08:35:24.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:24.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modified": "2021-11-19T08:35:14.953374+0000", 2021-11-19T08:35:24.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2021-11-19T08:35:24.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:24.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-19T08:35:24.675 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2021-11-19T08:35:24.675 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.667+0000 7f29cb83b700 1 -- 172.21.15.23:0/801581159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c40faae0 msgr2=0x7f29c410cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:24.675 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.667+0000 7f29cb83b700 1 --2- 172.21.15.23:0/801581159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c40faae0 0x7f29c410cbd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f29c000b1d0 tx=0x7f29c0006ee0).stop 2021-11-19T08:35:24.675 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.667+0000 7f29cb83b700 1 -- 172.21.15.23:0/801581159 shutdown_connections 2021-11-19T08:35:24.676 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.667+0000 7f29cb83b700 1 --2- 172.21.15.23:0/801581159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c40faae0 0x7f29c410cbd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:24.676 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.667+0000 7f29cb83b700 1 -- 172.21.15.23:0/801581159 >> 172.21.15.23:0/801581159 conn(0x7f29c40f6280 msgr2=0x7f29c406f830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:24.676 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.668+0000 7f29cb83b700 1 -- 172.21.15.23:0/801581159 shutdown_connections 2021-11-19T08:35:24.676 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:24.668+0000 7f29cb83b700 1 -- 172.21.15.23:0/801581159 wait complete. 2021-11-19T08:35:25.018 INFO:teuthology.orchestra.run.smithi023.stderr:mgr not available, waiting (2/15)... 2021-11-19T08:35:25.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:25 smithi023 conmon[33490]: audit 2021-11-19T08:35:24.667895+0000 mon.smithi023 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.23:0/801581159' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-19T08:35:27.020 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-19T08:35:27.235 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: cluster 2021-11-19T08:35:26.936803+0000 mon.smithi023 (mon 2021-11-19T08:35:27.236 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: .0) 9 : cluster [INF] Activating manager daemon smithi023.ehlpcl 2021-11-19T08:35:27.236 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: cluster 2021-11-19T08:35:26.939181+0000 mon.smithi023 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi023.ehlpcl(active, starting, since 0.00248736s) 2021-11-19T08:35:27.236 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: 2021-11-19T08:35:27.236 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: audit 2021-11-19T08 2021-11-19T08:35:27.237 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: :35:26.940289+0000 mon.smithi023 (mon.0) 11 : audit [DBG] 2021-11-19T08:35:27.237 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: from='mgr.14100 172.21.15.23:0/1770984284' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-19T08:35:27.237 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: audit 2021-11-19T08:35:26 2021-11-19T08:35:27.237 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: .940489+0000 mon.smithi023 (mon.0) 12 : audit 2021-11-19T08:35:27.237 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: [DBG] from='mgr.14100 172.21.15.23:0/1770984284' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-19T08:35:27.238 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: audit 2021-11-19T 2021-11-19T08:35:27.238 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: 08:35:26.940672+0000 mon.smithi023 (mon.0 2021-11-19T08:35:27.238 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: ) 13 : audit [DBG] from='mgr.14100 172.21.15.23:0/1770984284' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-19T08:35:27.238 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: audit 2021 2021-11-19T08:35:27.239 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: -11-19T08:35:26.940881+0000 2021-11-19T08:35:27.239 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: mon.smithi023 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.23:0/1770984284' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi023"}]: dispatch 2021-11-19T08:35:27.239 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: 2021-11-19T08:35:27.239 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:35:27.239 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: -19T08:35:26.941168+0000 mon.smithi023 ( 2021-11-19T08:35:27.240 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.23:0/1770984284' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mgr metadata", "who": "smithi023.ehlpcl", "id": "smithi023.ehlpcl"}]: dispatch 2021-11-19T08:35:27.240 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: cluster 2021-11-19T08:35:27.240 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: 2021-11-19T08:35:26.954477+0000 2021-11-19T08:35:27.240 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: mon.smithi023 (mon.0) 16 : cluster [INF] 2021-11-19T08:35:27.241 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: Manager daemon smithi023.ehlpcl is now available 2021-11-19T08:35:27.241 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:35:27.241 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: -19T08:35:26.963668+0000 mon.smithi023 (mon 2021-11-19T08:35:27.241 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: .0) 17 : audit [INF] from='mgr.14100 172.21.15.23:0/1770984284' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:27.241 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: audit 2021-11-19T08:35:27.242 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: 2021-11-19T08:35: 2021-11-19T08:35:27.242 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: 26.967283+0000 mon.smithi023 (mon.0) 18 : 2021-11-19T08:35:27.242 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: audit [INF] from='mgr.14100 172.21.15.23:0/1770984284' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:35:27.242 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: audit 2021-11-19T08:35:27.243 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: 2021-11-19T08:35:26.968550 2021-11-19T08:35:27.243 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 19 : audit [INF] 2021-11-19T08:35:27.243 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: from='mgr.14100 172.21.15.23:0/1770984284' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:35:27.243 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: audit 2021-11- 2021-11-19T08:35:27.243 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: 19T08:35:26.973902+0000 mon.smithi023 ( 2021-11-19T08:35:27.244 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.23:0/1770984284' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:27.244 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: audit 2021-11-19T08:35:27.244 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: 2021-11-19T08: 2021-11-19T08:35:27.244 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: 35:26.975978+0000 mon.smithi023 (mon.0) 21 : audit 2021-11-19T08:35:27.245 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:26 smithi023 conmon[33490]: [INF] from='mgr.14100 172.21.15.23:0/1770984284' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.540+0000 7fb222c7d700 1 Processor -- start 2021-11-19T08:35:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.540+0000 7fb222c7d700 1 -- start start 2021-11-19T08:35:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.540+0000 7fb222c7d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb21c0fd2a0 0x7fb21c0fd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:27.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.540+0000 7fb222c7d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c0fdbb0 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.540+0000 7fb220a19700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb21c0fd2a0 0x7fb21c0fd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:27.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.540+0000 7fb220a19700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb21c0fd2a0 0x7fb21c0fd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45956/0 (socket says 172.21.15.23:45956) 2021-11-19T08:35:27.534 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.540+0000 7fb220a19700 1 -- 172.21.15.23:0/3180380807 learned_addr learned my addr 172.21.15.23:0/3180380807 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:27.534 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.540+0000 7fb220a19700 1 -- 172.21.15.23:0/3180380807 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21c0fdcf0 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.534 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.541+0000 7fb220a19700 1 --2- 172.21.15.23:0/3180380807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb21c0fd2a0 0x7fb21c0fd6c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb21000b5f0 tx=0x7fb210006ee0).ready entity=mon.0 client_cookie=278b4c83b66c145f server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:27.534 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.541+0000 7fb21b7fe700 1 -- 172.21.15.23:0/3180380807 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb21000d600 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.535 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.541+0000 7fb21b7fe700 1 -- 172.21.15.23:0/3180380807 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb21000dbd0 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.535 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.541+0000 7fb21b7fe700 1 -- 172.21.15.23:0/3180380807 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb210011c00 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.535 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.541+0000 7fb222c7d700 1 -- 172.21.15.23:0/3180380807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb21c0fd2a0 msgr2=0x7fb21c0fd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:27.536 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.541+0000 7fb222c7d700 1 --2- 172.21.15.23:0/3180380807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb21c0fd2a0 0x7fb21c0fd6c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb21000b5f0 tx=0x7fb210006ee0).stop 2021-11-19T08:35:27.536 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.541+0000 7fb222c7d700 1 -- 172.21.15.23:0/3180380807 shutdown_connections 2021-11-19T08:35:27.537 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.541+0000 7fb222c7d700 1 --2- 172.21.15.23:0/3180380807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb21c0fd2a0 0x7fb21c0fd6c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:27.537 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.541+0000 7fb222c7d700 1 -- 172.21.15.23:0/3180380807 >> 172.21.15.23:0/3180380807 conn(0x7fb21c0f5a80 msgr2=0x7fb21c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:27.537 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.542+0000 7fb222c7d700 1 -- 172.21.15.23:0/3180380807 shutdown_connections 2021-11-19T08:35:27.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.542+0000 7fb222c7d700 1 -- 172.21.15.23:0/3180380807 wait complete. 2021-11-19T08:35:27.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.542+0000 7fb222c7d700 1 Processor -- start 2021-11-19T08:35:27.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.542+0000 7fb222c7d700 1 -- start start 2021-11-19T08:35:27.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.542+0000 7fb222c7d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb21c0fd2a0 0x7fb21c0fcb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.542+0000 7fb222c7d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c0faff0 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.543+0000 7fb220a19700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb21c0fd2a0 0x7fb21c0fcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.543+0000 7fb220a19700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb21c0fd2a0 0x7fb21c0fcb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45958/0 (socket says 172.21.15.23:45958) 2021-11-19T08:35:27.539 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.543+0000 7fb220a19700 1 -- 172.21.15.23:0/4004894104 learned_addr learned my addr 172.21.15.23:0/4004894104 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.543+0000 7fb220a19700 1 -- 172.21.15.23:0/4004894104 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21000b200 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.543+0000 7fb220a19700 1 --2- 172.21.15.23:0/4004894104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb21c0fd2a0 0x7fb21c0fcb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fb210006c70 tx=0x7fb210006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:27.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.543+0000 7fb2197fa700 1 -- 172.21.15.23:0/4004894104 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb210011690 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.543+0000 7fb2197fa700 1 -- 172.21.15.23:0/4004894104 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb21001b880 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.543+0000 7fb222c7d700 1 -- 172.21.15.23:0/4004894104 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21c0fb190 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.543+0000 7fb2197fa700 1 -- 172.21.15.23:0/4004894104 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb21001a650 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.541 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.543+0000 7fb222c7d700 1 -- 172.21.15.23:0/4004894104 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21c0fb5a0 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.544+0000 7fb2197fa700 1 -- 172.21.15.23:0/4004894104 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7fb21000f070 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.544+0000 7fb2197fa700 1 -- 172.21.15.23:0/4004894104 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb2100412b0 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.542 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.544+0000 7fb222c7d700 1 -- 172.21.15.23:0/4004894104 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21c045ad0 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.543 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.547+0000 7fb2197fa700 1 -- 172.21.15.23:0/4004894104 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb210016070 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.672+0000 7fb222c7d700 1 -- 172.21.15.23:0/4004894104 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb21c0fa660 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.672+0000 7fb2197fa700 1 -- 172.21.15.23:0/4004894104 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fb210025460 con 0x7fb21c0fd2a0 2021-11-19T08:35:27.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2021-11-19T08:35:27.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsid": "908b41ee-4913-11ec-8c2c-001a4aab830c", 2021-11-19T08:35:27.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "health": { 2021-11-19T08:35:27.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-19T08:35:27.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "checks": {}, 2021-11-19T08:35:27.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mutes": [] 2021-11-19T08:35:27.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:27.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-19T08:35:27.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum": [ 2021-11-19T08:35:27.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 0 2021-11-19T08:35:27.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2021-11-19T08:35:27.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-19T08:35:27.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "smithi023" 2021-11-19T08:35:27.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2021-11-19T08:35:27.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_age": 8, 2021-11-19T08:35:27.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "monmap": { 2021-11-19T08:35:27.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:27.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-19T08:35:27.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-19T08:35:27.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:27.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osdmap": { 2021-11-19T08:35:27.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:27.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-19T08:35:27.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-19T08:35:27.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-19T08:35:27.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-19T08:35:27.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-19T08:35:27.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-19T08:35:27.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:27.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgmap": { 2021-11-19T08:35:27.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-19T08:35:27.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-19T08:35:27.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-19T08:35:27.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-19T08:35:27.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-19T08:35:27.675 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-19T08:35:27.675 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-19T08:35:27.675 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-19T08:35:27.675 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:27.675 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsmap": { 2021-11-19T08:35:27.676 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:27.676 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-19T08:35:27.676 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-19T08:35:27.676 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:27.677 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-19T08:35:27.677 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "available": false, 2021-11-19T08:35:27.677 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-19T08:35:27.677 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modules": [ 2021-11-19T08:35:27.677 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "iostat", 2021-11-19T08:35:27.678 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "restful" 2021-11-19T08:35:27.678 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2021-11-19T08:35:27.678 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2021-11-19T08:35:27.678 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:27.678 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "servicemap": { 2021-11-19T08:35:27.679 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:27.679 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modified": "2021-11-19T08:35:14.953374+0000", 2021-11-19T08:35:27.679 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2021-11-19T08:35:27.679 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:27.680 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-19T08:35:27.680 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2021-11-19T08:35:27.680 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.673+0000 7fb222c7d700 1 -- 172.21.15.23:0/4004894104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb21c0fd2a0 msgr2=0x7fb21c0fcb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:27.680 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.673+0000 7fb222c7d700 1 --2- 172.21.15.23:0/4004894104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb21c0fd2a0 0x7fb21c0fcb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fb210006c70 tx=0x7fb210006ee0).stop 2021-11-19T08:35:27.681 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.673+0000 7fb222c7d700 1 -- 172.21.15.23:0/4004894104 shutdown_connections 2021-11-19T08:35:27.681 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.673+0000 7fb222c7d700 1 --2- 172.21.15.23:0/4004894104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb21c0fd2a0 0x7fb21c0fcb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:27.681 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.673+0000 7fb222c7d700 1 -- 172.21.15.23:0/4004894104 >> 172.21.15.23:0/4004894104 conn(0x7fb21c0f5a80 msgr2=0x7fb21c0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:27.681 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.673+0000 7fb222c7d700 1 -- 172.21.15.23:0/4004894104 shutdown_connections 2021-11-19T08:35:27.681 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:27.673+0000 7fb222c7d700 1 -- 172.21.15.23:0/4004894104 wait complete. 2021-11-19T08:35:27.956 INFO:teuthology.orchestra.run.smithi023.stderr:mgr not available, waiting (3/15)... 2021-11-19T08:35:28.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:27 smithi023 conmon[33490]: audit 2021-11-19T08:35:27.674186+0000 mon.smithi023 (mon.0) 22 : audit [DBG] from='client.? 172.21.15.23:0/4004894104' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-19T08:35:28.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:27 smithi023 conmon[33490]: cluster 2021-11-19T08:35:27.946448+0000 mon.smithi023 (mon.0) 23 : cluster [DBG] mgrmap e3: smithi023.ehlpcl(active, since 1.00975s) 2021-11-19T08:35:29.958 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2021-11-19T08:35:30.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:30 smithi023 conmon[33490]: cluster 2021-11-19T08:35:28.997056+0000 mon.smithi023 (mon. 2021-11-19T08:35:30.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:30 smithi023 conmon[33490]: 0) 24 : cluster [DBG] mgrmap e4: smithi023.ehlpcl(active, since 2s) 2021-11-19T08:35:32.179 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.142+0000 7f8e5f8a5700 1 Processor -- start 2021-11-19T08:35:32.180 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.142+0000 7f8e5f8a5700 1 -- start start 2021-11-19T08:35:32.180 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.142+0000 7f8e5f8a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e580fc1d0 0x7f8e580fc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:32.180 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.142+0000 7f8e5f8a5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580fcae0 con 0x7f8e580fc1d0 2021-11-19T08:35:32.181 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.142+0000 7f8e5d641700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e580fc1d0 0x7f8e580fc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:32.181 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.142+0000 7f8e5d641700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e580fc1d0 0x7f8e580fc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45970/0 (socket says 172.21.15.23:45970) 2021-11-19T08:35:32.182 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.142+0000 7f8e5d641700 1 -- 172.21.15.23:0/2918790085 learned_addr learned my addr 172.21.15.23:0/2918790085 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:32.182 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.143+0000 7f8e5d641700 1 -- 172.21.15.23:0/2918790085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e580fcc20 con 0x7f8e580fc1d0 2021-11-19T08:35:32.182 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.143+0000 7f8e5d641700 1 --2- 172.21.15.23:0/2918790085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e580fc1d0 0x7f8e580fc5f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f8e5400b5f0 tx=0x7f8e54006ee0).ready entity=mon.0 client_cookie=e75d80d0b9f5ad3a server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:32.182 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.143+0000 7f8e47fff700 1 -- 172.21.15.23:0/2918790085 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e5400d600 con 0x7f8e580fc1d0 2021-11-19T08:35:32.183 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.143+0000 7f8e47fff700 1 -- 172.21.15.23:0/2918790085 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8e5400dbd0 con 0x7f8e580fc1d0 2021-11-19T08:35:32.187 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.143+0000 7f8e47fff700 1 -- 172.21.15.23:0/2918790085 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e54011c00 con 0x7f8e580fc1d0 2021-11-19T08:35:32.188 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.143+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2918790085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e580fc1d0 msgr2=0x7f8e580fc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:32.188 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.143+0000 7f8e5f8a5700 1 --2- 172.21.15.23:0/2918790085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e580fc1d0 0x7f8e580fc5f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f8e5400b5f0 tx=0x7f8e54006ee0).stop 2021-11-19T08:35:32.188 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.144+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2918790085 shutdown_connections 2021-11-19T08:35:32.188 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.144+0000 7f8e5f8a5700 1 --2- 172.21.15.23:0/2918790085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e580fc1d0 0x7f8e580fc5f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:32.189 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.144+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2918790085 >> 172.21.15.23:0/2918790085 conn(0x7f8e580f5a80 msgr2=0x7f8e580f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:32.189 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.144+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2918790085 shutdown_connections 2021-11-19T08:35:32.189 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.144+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2918790085 wait complete. 2021-11-19T08:35:32.189 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.144+0000 7f8e5f8a5700 1 Processor -- start 2021-11-19T08:35:32.190 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.144+0000 7f8e5f8a5700 1 -- start start 2021-11-19T08:35:32.190 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.144+0000 7f8e5f8a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e580fc1d0 0x7f8e5810cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:32.191 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.144+0000 7f8e5f8a5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5810d100 con 0x7f8e580fc1d0 2021-11-19T08:35:32.191 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.144+0000 7f8e5d641700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e580fc1d0 0x7f8e5810cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:32.191 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.144+0000 7f8e5d641700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e580fc1d0 0x7f8e5810cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45972/0 (socket says 172.21.15.23:45972) 2021-11-19T08:35:32.192 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.145+0000 7f8e5d641700 1 -- 172.21.15.23:0/2758302906 learned_addr learned my addr 172.21.15.23:0/2758302906 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:32.193 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.145+0000 7f8e5d641700 1 -- 172.21.15.23:0/2758302906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5400b200 con 0x7f8e580fc1d0 2021-11-19T08:35:32.193 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.145+0000 7f8e5d641700 1 --2- 172.21.15.23:0/2758302906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e580fc1d0 0x7f8e5810cc10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f8e54000c00 tx=0x7f8e54006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:32.193 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.145+0000 7f8e45ffb700 1 -- 172.21.15.23:0/2758302906 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e54011690 con 0x7f8e580fc1d0 2021-11-19T08:35:32.194 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.145+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2758302906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5810d300 con 0x7f8e580fc1d0 2021-11-19T08:35:32.194 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.145+0000 7f8e45ffb700 1 -- 172.21.15.23:0/2758302906 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8e5400d9f0 con 0x7f8e580fc1d0 2021-11-19T08:35:32.194 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.145+0000 7f8e45ffb700 1 -- 172.21.15.23:0/2758302906 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e5401b710 con 0x7f8e580fc1d0 2021-11-19T08:35:32.195 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.145+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2758302906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5810d710 con 0x7f8e580fc1d0 2021-11-19T08:35:32.195 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.146+0000 7f8e45ffb700 1 -- 172.21.15.23:0/2758302906 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f8e5400f070 con 0x7f8e580fc1d0 2021-11-19T08:35:32.195 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.146+0000 7f8e45ffb700 1 --2- 172.21.15.23:0/2758302906 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7f8e4002fd50 0x7f8e40032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:32.196 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.146+0000 7f8e45ffb700 1 -- 172.21.15.23:0/2758302906 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8e540410f0 con 0x7f8e580fc1d0 2021-11-19T08:35:32.196 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.146+0000 7f8e5ce40700 1 --2- 172.21.15.23:0/2758302906 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7f8e4002fd50 0x7f8e40032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:32.197 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.146+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2758302906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e58059100 con 0x7f8e580fc1d0 2021-11-19T08:35:32.197 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.147+0000 7f8e5ce40700 1 --2- 172.21.15.23:0/2758302906 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7f8e4002fd50 0x7f8e40032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f8e480096a0 tx=0x7f8e48006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:32.197 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.151+0000 7f8e45ffb700 1 -- 172.21.15.23:0/2758302906 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e54016070 con 0x7f8e580fc1d0 2021-11-19T08:35:32.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.549+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2758302906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8e58101d30 con 0x7f8e580fc1d0 2021-11-19T08:35:32.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.550+0000 7f8e45ffb700 1 -- 172.21.15.23:0/2758302906 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f8e540226d0 con 0x7f8e580fc1d0 2021-11-19T08:35:32.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.539 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2021-11-19T08:35:32.539 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsid": "908b41ee-4913-11ec-8c2c-001a4aab830c", 2021-11-19T08:35:32.539 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "health": { 2021-11-19T08:35:32.539 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-11-19T08:35:32.539 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "checks": {}, 2021-11-19T08:35:32.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mutes": [] 2021-11-19T08:35:32.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:32.548 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-11-19T08:35:32.548 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum": [ 2021-11-19T08:35:32.549 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 0 2021-11-19T08:35:32.549 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2021-11-19T08:35:32.549 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_names": [ 2021-11-19T08:35:32.549 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "smithi023" 2021-11-19T08:35:32.550 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2021-11-19T08:35:32.550 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_age": 13, 2021-11-19T08:35:32.550 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "monmap": { 2021-11-19T08:35:32.550 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:32.551 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2021-11-19T08:35:32.551 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_mons": 1 2021-11-19T08:35:32.551 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:32.552 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osdmap": { 2021-11-19T08:35:32.552 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:32.552 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_osds": 0, 2021-11-19T08:35:32.552 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-11-19T08:35:32.553 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-11-19T08:35:32.553 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-11-19T08:35:32.553 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-11-19T08:35:32.553 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-11-19T08:35:32.553 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:32.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgmap": { 2021-11-19T08:35:32.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-11-19T08:35:32.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-11-19T08:35:32.555 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pools": 0, 2021-11-19T08:35:32.555 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_objects": 0, 2021-11-19T08:35:32.555 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-11-19T08:35:32.556 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-11-19T08:35:32.556 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-11-19T08:35:32.556 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_total": 0 2021-11-19T08:35:32.557 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:32.557 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsmap": { 2021-11-19T08:35:32.557 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:32.557 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "by_rank": [], 2021-11-19T08:35:32.558 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "up:standby": 0 2021-11-19T08:35:32.558 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:32.558 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mgrmap": { 2021-11-19T08:35:32.558 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "available": true, 2021-11-19T08:35:32.558 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-11-19T08:35:32.559 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modules": [ 2021-11-19T08:35:32.559 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "iostat", 2021-11-19T08:35:32.559 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "restful" 2021-11-19T08:35:32.559 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2021-11-19T08:35:32.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2021-11-19T08:35:32.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:32.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "servicemap": { 2021-11-19T08:35:32.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2021-11-19T08:35:32.561 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modified": "2021-11-19T08:35:14.953374+0000", 2021-11-19T08:35:32.561 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2021-11-19T08:35:32.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2021-11-19T08:35:32.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "progress_events": {} 2021-11-19T08:35:32.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2021-11-19T08:35:32.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.552+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2758302906 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7f8e4002fd50 msgr2=0x7f8e40032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:32.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.552+0000 7f8e5f8a5700 1 --2- 172.21.15.23:0/2758302906 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7f8e4002fd50 0x7f8e40032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f8e480096a0 tx=0x7f8e48006b40).stop 2021-11-19T08:35:32.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.552+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2758302906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e580fc1d0 msgr2=0x7f8e5810cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:32.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.552+0000 7f8e5f8a5700 1 --2- 172.21.15.23:0/2758302906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e580fc1d0 0x7f8e5810cc10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f8e54000c00 tx=0x7f8e54006ee0).stop 2021-11-19T08:35:32.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.552+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2758302906 shutdown_connections 2021-11-19T08:35:32.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.552+0000 7f8e5f8a5700 1 --2- 172.21.15.23:0/2758302906 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7f8e4002fd50 0x7f8e40032210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:32.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.552+0000 7f8e5f8a5700 1 --2- 172.21.15.23:0/2758302906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e580fc1d0 0x7f8e5810cc10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:32.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.552+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2758302906 >> 172.21.15.23:0/2758302906 conn(0x7f8e580f5a80 msgr2=0x7f8e580f75d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:32.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.552+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2758302906 shutdown_connections 2021-11-19T08:35:32.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:32.552+0000 7f8e5f8a5700 1 -- 172.21.15.23:0/2758302906 wait complete. 2021-11-19T08:35:32.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:32 smithi023 conmon[33490]: audit 2021-11-19T08 2021-11-19T08:35:32.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:32 smithi023 conmon[33490]: :35:32.552142+0000 mon.smithi023 (mon.0) 25 : audit [DBG] from='client.? 172.21.15.23:0/2758302906' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-11-19T08:35:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:mgr is available 2021-11-19T08:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:Enabling cephadm module... 2021-11-19T08:35:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2021-11-19T08:35:36.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.187+0000 7fc17b3e6700 1 Processor -- start 2021-11-19T08:35:36.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.187+0000 7fc17b3e6700 1 -- start start 2021-11-19T08:35:36.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.187+0000 7fc17b3e6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1740fd280 0x7fc1740fd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:36.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.187+0000 7fc17b3e6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1740fdb90 con 0x7fc1740fd280 2021-11-19T08:35:36.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.187+0000 7fc179182700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1740fd280 0x7fc1740fd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:36.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.187+0000 7fc179182700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1740fd280 0x7fc1740fd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45976/0 (socket says 172.21.15.23:45976) 2021-11-19T08:35:36.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.187+0000 7fc179182700 1 -- 172.21.15.23:0/3654439654 learned_addr learned my addr 172.21.15.23:0/3654439654 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:36.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.188+0000 7fc179182700 1 -- 172.21.15.23:0/3654439654 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1740fdcd0 con 0x7fc1740fd280 2021-11-19T08:35:36.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.188+0000 7fc179182700 1 --2- 172.21.15.23:0/3654439654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1740fd280 0x7fc1740fd6a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fc17000b5f0 tx=0x7fc170006ee0).ready entity=mon.0 client_cookie=8c54f31588f4a034 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:36.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.188+0000 7fc163fff700 1 -- 172.21.15.23:0/3654439654 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc17000d600 con 0x7fc1740fd280 2021-11-19T08:35:36.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.188+0000 7fc163fff700 1 -- 172.21.15.23:0/3654439654 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc17000dbd0 con 0x7fc1740fd280 2021-11-19T08:35:36.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.188+0000 7fc163fff700 1 -- 172.21.15.23:0/3654439654 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc170011c00 con 0x7fc1740fd280 2021-11-19T08:35:36.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.188+0000 7fc17b3e6700 1 -- 172.21.15.23:0/3654439654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1740fd280 msgr2=0x7fc1740fd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:36.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.188+0000 7fc17b3e6700 1 --2- 172.21.15.23:0/3654439654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1740fd280 0x7fc1740fd6a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fc17000b5f0 tx=0x7fc170006ee0).stop 2021-11-19T08:35:36.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.188+0000 7fc17b3e6700 1 -- 172.21.15.23:0/3654439654 shutdown_connections 2021-11-19T08:35:36.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.188+0000 7fc17b3e6700 1 --2- 172.21.15.23:0/3654439654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1740fd280 0x7fc1740fd6a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:36.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.188+0000 7fc17b3e6700 1 -- 172.21.15.23:0/3654439654 >> 172.21.15.23:0/3654439654 conn(0x7fc1740f5a80 msgr2=0x7fc1740f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:36.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.189+0000 7fc17b3e6700 1 -- 172.21.15.23:0/3654439654 shutdown_connections 2021-11-19T08:35:36.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.189+0000 7fc17b3e6700 1 -- 172.21.15.23:0/3654439654 wait complete. 2021-11-19T08:35:36.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.189+0000 7fc17b3e6700 1 Processor -- start 2021-11-19T08:35:36.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.189+0000 7fc17b3e6700 1 -- start start 2021-11-19T08:35:36.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.189+0000 7fc17b3e6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1740fd280 0x7fc1740fcb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:36.334 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.189+0000 7fc17b3e6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1740fafd0 con 0x7fc1740fd280 2021-11-19T08:35:36.334 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.189+0000 7fc179182700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1740fd280 0x7fc1740fcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:36.334 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.189+0000 7fc179182700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1740fd280 0x7fc1740fcb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45978/0 (socket says 172.21.15.23:45978) 2021-11-19T08:35:36.334 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.189+0000 7fc179182700 1 -- 172.21.15.23:0/1914089129 learned_addr learned my addr 172.21.15.23:0/1914089129 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:36.335 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.190+0000 7fc179182700 1 -- 172.21.15.23:0/1914089129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17000b200 con 0x7fc1740fd280 2021-11-19T08:35:36.335 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.190+0000 7fc179182700 1 --2- 172.21.15.23:0/1914089129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1740fd280 0x7fc1740fcb50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fc170002f40 tx=0x7fc170006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:36.335 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.190+0000 7fc161ffb700 1 -- 172.21.15.23:0/1914089129 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc17001b410 con 0x7fc1740fd280 2021-11-19T08:35:36.336 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.190+0000 7fc161ffb700 1 -- 172.21.15.23:0/1914089129 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc17001b9e0 con 0x7fc1740fd280 2021-11-19T08:35:36.336 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.190+0000 7fc17b3e6700 1 -- 172.21.15.23:0/1914089129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1740fb170 con 0x7fc1740fd280 2021-11-19T08:35:36.336 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.190+0000 7fc161ffb700 1 -- 172.21.15.23:0/1914089129 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc17001a6e0 con 0x7fc1740fd280 2021-11-19T08:35:36.336 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.190+0000 7fc17b3e6700 1 -- 172.21.15.23:0/1914089129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1740fb580 con 0x7fc1740fd280 2021-11-19T08:35:36.337 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.191+0000 7fc161ffb700 1 -- 172.21.15.23:0/1914089129 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fc17000f070 con 0x7fc1740fd280 2021-11-19T08:35:36.337 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.191+0000 7fc161ffb700 1 --2- 172.21.15.23:0/1914089129 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7fc15c02fd50 0x7fc15c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:36.337 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.191+0000 7fc161ffb700 1 -- 172.21.15.23:0/1914089129 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fc170040ff0 con 0x7fc1740fd280 2021-11-19T08:35:36.337 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.191+0000 7fc178981700 1 --2- 172.21.15.23:0/1914089129 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7fc15c02fd50 0x7fc15c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:36.338 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.191+0000 7fc17b3e6700 1 -- 172.21.15.23:0/1914089129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc174059100 con 0x7fc1740fd280 2021-11-19T08:35:36.339 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.192+0000 7fc178981700 1 --2- 172.21.15.23:0/1914089129 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7fc15c02fd50 0x7fc15c032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc1640096a0 tx=0x7fc164006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:36.340 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.196+0000 7fc161ffb700 1 -- 172.21.15.23:0/1914089129 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc170016070 con 0x7fc1740fd280 2021-11-19T08:35:36.541 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.555+0000 7fc17b3e6700 1 -- 172.21.15.23:0/1914089129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fc174045ad0 con 0x7fc1740fd280 2021-11-19T08:35:36.594 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.608+0000 7fc161ffb700 1 -- 172.21.15.23:0/1914089129 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc17001a840 con 0x7fc1740fd280 2021-11-19T08:35:36.595 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.609+0000 7fc161ffb700 1 -- 172.21.15.23:0/1914089129 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fc17002fa90 con 0x7fc1740fd280 2021-11-19T08:35:36.599 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.611+0000 7fc17b3e6700 1 -- 172.21.15.23:0/1914089129 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7fc15c02fd50 msgr2=0x7fc15c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:36.599 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.611+0000 7fc17b3e6700 1 --2- 172.21.15.23:0/1914089129 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7fc15c02fd50 0x7fc15c032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc1640096a0 tx=0x7fc164006b40).stop 2021-11-19T08:35:36.599 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.612+0000 7fc17b3e6700 1 -- 172.21.15.23:0/1914089129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1740fd280 msgr2=0x7fc1740fcb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:36.599 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.612+0000 7fc17b3e6700 1 --2- 172.21.15.23:0/1914089129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1740fd280 0x7fc1740fcb50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fc170002f40 tx=0x7fc170006ee0).stop 2021-11-19T08:35:36.600 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.612+0000 7fc17b3e6700 1 -- 172.21.15.23:0/1914089129 shutdown_connections 2021-11-19T08:35:36.600 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.612+0000 7fc17b3e6700 1 --2- 172.21.15.23:0/1914089129 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7fc15c02fd50 0x7fc15c032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:36.600 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.612+0000 7fc17b3e6700 1 --2- 172.21.15.23:0/1914089129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1740fd280 0x7fc1740fcb50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:36.600 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.612+0000 7fc17b3e6700 1 -- 172.21.15.23:0/1914089129 >> 172.21.15.23:0/1914089129 conn(0x7fc1740f5a80 msgr2=0x7fc1740ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:36.600 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.612+0000 7fc17b3e6700 1 -- 172.21.15.23:0/1914089129 shutdown_connections 2021-11-19T08:35:36.601 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:36.612+0000 7fc17b3e6700 1 -- 172.21.15.23:0/1914089129 wait complete. 2021-11-19T08:35:36.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:36 smithi023 conmon[33490]: audit 2021-11-19T08:35:36.556803+0000 mon.smithi023 ( 2021-11-19T08:35:36.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:36 smithi023 conmon[33490]: mon.0) 26 : audit [INF] from='client.? 172.21.15.23:0/1914089129' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2021-11-19T08:35:37.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:37 smithi023 conmon[33490]: audit 2021-11-19T08:35:36.610740+0000 mon.smithi023 ( 2021-11-19T08:35:37.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:37 smithi023 conmon[33490]: mon.0) 27 : audit [INF] from='client.? 172.21.15.23:0/1914089129' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2021-11-19T08:35:37.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:37 smithi023 conmon[33490]: cluster 2021-11-19T08:35:36.610801+0000 mon.smithi023 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi023.ehlpcl(active, since 9s) 2021-11-19T08:35:38.359 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2021-11-19T08:35:40.478 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.488+0000 7ff38504b700 1 Processor -- start 2021-11-19T08:35:40.479 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.488+0000 7ff38504b700 1 -- start start 2021-11-19T08:35:40.480 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.488+0000 7ff38504b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38006fcd0 0x7ff3800fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:40.481 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.488+0000 7ff38504b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800fed20 con 0x7ff38006fcd0 2021-11-19T08:35:40.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.488+0000 7ff37ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38006fcd0 0x7ff3800fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:40.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.488+0000 7ff37ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38006fcd0 0x7ff3800fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45996/0 (socket says 172.21.15.23:45996) 2021-11-19T08:35:40.483 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.488+0000 7ff37ed9d700 1 -- 172.21.15.23:0/1326830039 learned_addr learned my addr 172.21.15.23:0/1326830039 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:40.483 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.489+0000 7ff37ed9d700 1 -- 172.21.15.23:0/1326830039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3800fee60 con 0x7ff38006fcd0 2021-11-19T08:35:40.483 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.489+0000 7ff37ed9d700 1 --2- 172.21.15.23:0/1326830039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38006fcd0 0x7ff3800fe7a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff37000b5f0 tx=0x7ff370006ee0).ready entity=mon.0 client_cookie=6220f4f0badf8beb server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:40.484 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.489+0000 7ff37dd9b700 1 -- 172.21.15.23:0/1326830039 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff37000d600 con 0x7ff38006fcd0 2021-11-19T08:35:40.484 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.489+0000 7ff37dd9b700 1 -- 172.21.15.23:0/1326830039 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff37000dbd0 con 0x7ff38006fcd0 2021-11-19T08:35:40.485 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.489+0000 7ff37dd9b700 1 -- 172.21.15.23:0/1326830039 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff370011c00 con 0x7ff38006fcd0 2021-11-19T08:35:40.485 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.489+0000 7ff38504b700 1 -- 172.21.15.23:0/1326830039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38006fcd0 msgr2=0x7ff3800fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:40.486 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.489+0000 7ff38504b700 1 --2- 172.21.15.23:0/1326830039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38006fcd0 0x7ff3800fe7a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff37000b5f0 tx=0x7ff370006ee0).stop 2021-11-19T08:35:40.486 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.490+0000 7ff38504b700 1 -- 172.21.15.23:0/1326830039 shutdown_connections 2021-11-19T08:35:40.487 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.490+0000 7ff38504b700 1 --2- 172.21.15.23:0/1326830039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38006fcd0 0x7ff3800fe7a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:40.487 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.490+0000 7ff38504b700 1 -- 172.21.15.23:0/1326830039 >> 172.21.15.23:0/1326830039 conn(0x7ff3800f6280 msgr2=0x7ff3800f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:40.488 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.490+0000 7ff38504b700 1 -- 172.21.15.23:0/1326830039 shutdown_connections 2021-11-19T08:35:40.488 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.490+0000 7ff38504b700 1 -- 172.21.15.23:0/1326830039 wait complete. 2021-11-19T08:35:40.488 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.490+0000 7ff38504b700 1 Processor -- start 2021-11-19T08:35:40.489 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.490+0000 7ff38504b700 1 -- start start 2021-11-19T08:35:40.489 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.491+0000 7ff38504b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38006fcd0 0x7ff3800fc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:40.489 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.491+0000 7ff38504b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800fa6a0 con 0x7ff38006fcd0 2021-11-19T08:35:40.489 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.491+0000 7ff37ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38006fcd0 0x7ff3800fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:40.490 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.491+0000 7ff37ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38006fcd0 0x7ff3800fc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45998/0 (socket says 172.21.15.23:45998) 2021-11-19T08:35:40.490 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.491+0000 7ff37ed9d700 1 -- 172.21.15.23:0/3303192771 learned_addr learned my addr 172.21.15.23:0/3303192771 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:40.490 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.491+0000 7ff37ed9d700 1 -- 172.21.15.23:0/3303192771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37000b200 con 0x7ff38006fcd0 2021-11-19T08:35:40.491 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.491+0000 7ff37ed9d700 1 --2- 172.21.15.23:0/3303192771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38006fcd0 0x7ff3800fc220 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff37000e040 tx=0x7ff370006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:40.491 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.491+0000 7ff3777fe700 1 -- 172.21.15.23:0/3303192771 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff37001b410 con 0x7ff38006fcd0 2021-11-19T08:35:40.491 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.491+0000 7ff38504b700 1 -- 172.21.15.23:0/3303192771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3800fa840 con 0x7ff38006fcd0 2021-11-19T08:35:40.491 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.491+0000 7ff38504b700 1 -- 172.21.15.23:0/3303192771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3800fac50 con 0x7ff38006fcd0 2021-11-19T08:35:40.491 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.492+0000 7ff3777fe700 1 -- 172.21.15.23:0/3303192771 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff37001b9e0 con 0x7ff38006fcd0 2021-11-19T08:35:40.492 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.492+0000 7ff3777fe700 1 -- 172.21.15.23:0/3303192771 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff37001a770 con 0x7ff38006fcd0 2021-11-19T08:35:40.493 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.493+0000 7ff3777fe700 1 -- 172.21.15.23:0/3303192771 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff37000f070 con 0x7ff38006fcd0 2021-11-19T08:35:40.493 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.493+0000 7ff38504b700 1 -- 172.21.15.23:0/3303192771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff364004fa0 con 0x7ff38006fcd0 2021-11-19T08:35:40.493 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.493+0000 7ff3777fe700 1 --2- 172.21.15.23:0/3303192771 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7ff36002fd50 0x7ff360032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:40.493 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.493+0000 7ff3777fe700 1 -- 172.21.15.23:0/3303192771 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff370040fe0 con 0x7ff38006fcd0 2021-11-19T08:35:40.494 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.493+0000 7ff37e59c700 1 -- 172.21.15.23:0/3303192771 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7ff36002fd50 msgr2=0x7ff360032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/945698991 2021-11-19T08:35:40.494 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.493+0000 7ff37e59c700 1 --2- 172.21.15.23:0/3303192771 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7ff36002fd50 0x7ff360032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-19T08:35:40.494 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.497+0000 7ff3777fe700 1 -- 172.21.15.23:0/3303192771 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff37001a9e0 con 0x7ff38006fcd0 2021-11-19T08:35:40.680 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.694+0000 7ff37e59c700 1 -- 172.21.15.23:0/3303192771 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7ff36002fd50 msgr2=0x7ff360032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/945698991 2021-11-19T08:35:40.681 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.694+0000 7ff37e59c700 1 --2- 172.21.15.23:0/3303192771 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7ff36002fd50 0x7ff360032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-19T08:35:40.846 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.859+0000 7ff38504b700 1 -- 172.21.15.23:0/3303192771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ff364005e80 con 0x7ff38006fcd0 2021-11-19T08:35:40.846 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.859+0000 7ff3777fe700 1 -- 172.21.15.23:0/3303192771 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7ff370016070 con 0x7ff38006fcd0 2021-11-19T08:35:40.847 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2021-11-19T08:35:40.847 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 5, 2021-11-19T08:35:40.847 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "available": true, 2021-11-19T08:35:40.847 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "active_name": "smithi023.ehlpcl", 2021-11-19T08:35:40.848 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_standby": 0 2021-11-19T08:35:40.848 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2021-11-19T08:35:40.849 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.862+0000 7ff38504b700 1 -- 172.21.15.23:0/3303192771 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7ff36002fd50 msgr2=0x7ff360032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-19T08:35:40.850 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.862+0000 7ff38504b700 1 --2- 172.21.15.23:0/3303192771 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7ff36002fd50 0x7ff360032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:35:40.850 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.862+0000 7ff38504b700 1 -- 172.21.15.23:0/3303192771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38006fcd0 msgr2=0x7ff3800fc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:40.850 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.862+0000 7ff38504b700 1 --2- 172.21.15.23:0/3303192771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38006fcd0 0x7ff3800fc220 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff37000e040 tx=0x7ff370006ee0).stop 2021-11-19T08:35:40.850 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.862+0000 7ff38504b700 1 -- 172.21.15.23:0/3303192771 shutdown_connections 2021-11-19T08:35:40.851 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.862+0000 7ff38504b700 1 --2- 172.21.15.23:0/3303192771 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7ff36002fd50 0x7ff360032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:35:40.851 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.862+0000 7ff38504b700 1 --2- 172.21.15.23:0/3303192771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff38006fcd0 0x7ff3800fc220 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:40.851 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.862+0000 7ff38504b700 1 -- 172.21.15.23:0/3303192771 >> 172.21.15.23:0/3303192771 conn(0x7ff3800f6280 msgr2=0x7ff3800f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:40.851 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.862+0000 7ff38504b700 1 -- 172.21.15.23:0/3303192771 shutdown_connections 2021-11-19T08:35:40.851 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:40.862+0000 7ff38504b700 1 -- 172.21.15.23:0/3303192771 wait complete. 2021-11-19T08:35:41.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:40 smithi023 conmon[33490]: audit 2021-11-19T08:35:40.860559 2021-11-19T08:35:41.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:40 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.23:0/3303192771' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-11-19T08:35:41.208 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for the mgr to restart... 2021-11-19T08:35:41.209 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for mgr epoch 5... 2021-11-19T08:35:41.209 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2021-11-19T08:35:41.715 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.728+0000 7faa4bf8f700 1 Processor -- start 2021-11-19T08:35:41.718 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.729+0000 7faa4bf8f700 1 -- start start 2021-11-19T08:35:41.719 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.729+0000 7faa4bf8f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa4406fcd0 0x7faa440fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:41.719 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.729+0000 7faa4bf8f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa440fed20 con 0x7faa4406fcd0 2021-11-19T08:35:41.719 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.729+0000 7faa49d2b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa4406fcd0 0x7faa440fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:41.719 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.729+0000 7faa49d2b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa4406fcd0 0x7faa440fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46004/0 (socket says 172.21.15.23:46004) 2021-11-19T08:35:41.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.729+0000 7faa49d2b700 1 -- 172.21.15.23:0/1592820326 learned_addr learned my addr 172.21.15.23:0/1592820326 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:41.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.729+0000 7faa49d2b700 1 -- 172.21.15.23:0/1592820326 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa440fee60 con 0x7faa4406fcd0 2021-11-19T08:35:41.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.729+0000 7faa49d2b700 1 --2- 172.21.15.23:0/1592820326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa4406fcd0 0x7faa440fe7a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7faa3800b5f0 tx=0x7faa38006ee0).ready entity=mon.0 client_cookie=ae569b8636a146c0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:41.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.730+0000 7faa48d29700 1 -- 172.21.15.23:0/1592820326 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa3800d600 con 0x7faa4406fcd0 2021-11-19T08:35:41.721 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.730+0000 7faa48d29700 1 -- 172.21.15.23:0/1592820326 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7faa3800dbd0 con 0x7faa4406fcd0 2021-11-19T08:35:41.721 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.730+0000 7faa48d29700 1 -- 172.21.15.23:0/1592820326 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa38011c00 con 0x7faa4406fcd0 2021-11-19T08:35:41.721 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.730+0000 7faa4bf8f700 1 -- 172.21.15.23:0/1592820326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa4406fcd0 msgr2=0x7faa440fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:41.721 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.730+0000 7faa4bf8f700 1 --2- 172.21.15.23:0/1592820326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa4406fcd0 0x7faa440fe7a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7faa3800b5f0 tx=0x7faa38006ee0).stop 2021-11-19T08:35:41.721 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.730+0000 7faa4bf8f700 1 -- 172.21.15.23:0/1592820326 shutdown_connections 2021-11-19T08:35:41.722 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.730+0000 7faa4bf8f700 1 --2- 172.21.15.23:0/1592820326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa4406fcd0 0x7faa440fe7a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:41.722 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.730+0000 7faa4bf8f700 1 -- 172.21.15.23:0/1592820326 >> 172.21.15.23:0/1592820326 conn(0x7faa440f6280 msgr2=0x7faa440f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:41.722 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.731+0000 7faa4bf8f700 1 -- 172.21.15.23:0/1592820326 shutdown_connections 2021-11-19T08:35:41.722 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.731+0000 7faa4bf8f700 1 -- 172.21.15.23:0/1592820326 wait complete. 2021-11-19T08:35:41.723 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.731+0000 7faa4bf8f700 1 Processor -- start 2021-11-19T08:35:41.723 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.731+0000 7faa4bf8f700 1 -- start start 2021-11-19T08:35:41.723 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.731+0000 7faa4bf8f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa4406fcd0 0x7faa440fc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:41.723 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.731+0000 7faa4bf8f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa440fa6a0 con 0x7faa4406fcd0 2021-11-19T08:35:41.723 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.732+0000 7faa49d2b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa4406fcd0 0x7faa440fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:41.725 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.732+0000 7faa49d2b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa4406fcd0 0x7faa440fc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46006/0 (socket says 172.21.15.23:46006) 2021-11-19T08:35:41.725 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.732+0000 7faa49d2b700 1 -- 172.21.15.23:0/771401678 learned_addr learned my addr 172.21.15.23:0/771401678 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:41.726 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.732+0000 7faa49d2b700 1 -- 172.21.15.23:0/771401678 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3800b200 con 0x7faa4406fcd0 2021-11-19T08:35:41.726 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.732+0000 7faa49d2b700 1 --2- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa4406fcd0 0x7faa440fc220 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7faa38000c00 tx=0x7faa38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:41.727 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.732+0000 7faa367fc700 1 -- 172.21.15.23:0/771401678 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa38011690 con 0x7faa4406fcd0 2021-11-19T08:35:41.727 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.732+0000 7faa367fc700 1 -- 172.21.15.23:0/771401678 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7faa3800d9f0 con 0x7faa4406fcd0 2021-11-19T08:35:41.728 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.732+0000 7faa4bf8f700 1 -- 172.21.15.23:0/771401678 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa440fa840 con 0x7faa4406fcd0 2021-11-19T08:35:41.728 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.732+0000 7faa367fc700 1 -- 172.21.15.23:0/771401678 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa3801b710 con 0x7faa4406fcd0 2021-11-19T08:35:41.729 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.732+0000 7faa4bf8f700 1 -- 172.21.15.23:0/771401678 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa440fac50 con 0x7faa4406fcd0 2021-11-19T08:35:41.729 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.733+0000 7faa367fc700 1 -- 172.21.15.23:0/771401678 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7faa3800f070 con 0x7faa4406fcd0 2021-11-19T08:35:41.729 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.733+0000 7faa367fc700 1 --2- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7faa2c02fdb0 0x7faa2c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:41.730 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.733+0000 7faa367fc700 1 -- 172.21.15.23:0/771401678 --> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa2c032930 con 0x7faa2c02fdb0 2021-11-19T08:35:41.730 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.733+0000 7faa367fc700 1 -- 172.21.15.23:0/771401678 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7faa38041050 con 0x7faa4406fcd0 2021-11-19T08:35:41.731 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.733+0000 7faa4952a700 1 -- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7faa2c02fdb0 msgr2=0x7faa2c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/945698991 2021-11-19T08:35:41.731 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.733+0000 7faa4952a700 1 --2- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7faa2c02fdb0 0x7faa2c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-19T08:35:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.934+0000 7faa4952a700 1 -- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7faa2c02fdb0 msgr2=0x7faa2c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/945698991 2021-11-19T08:35:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:41.934+0000 7faa4952a700 1 --2- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7faa2c02fdb0 0x7faa2c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-19T08:35:42.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:42.335+0000 7faa4952a700 1 -- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7faa2c02fdb0 msgr2=0x7faa2c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/945698991 2021-11-19T08:35:42.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:42.335+0000 7faa4952a700 1 --2- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7faa2c02fdb0 0x7faa2c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-19T08:35:42.585 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:42.598+0000 7faa367fc700 1 -- 172.21.15.23:0/771401678 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7faa3801aaf0 con 0x7faa4406fcd0 2021-11-19T08:35:42.585 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:42.598+0000 7faa367fc700 1 -- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7faa2c02fdb0 msgr2=0x7faa2c032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-19T08:35:42.585 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:42.598+0000 7faa367fc700 1 --2- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7faa2c02fdb0 0x7faa2c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:35:42.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:42 smithi023 conmon[33490]: cluster 2021-11-19T08:35:42.544878+0000 mon.smithi023 (mon 2021-11-19T08:35:42.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:42 smithi023 conmon[33490]: .0) 30 : cluster [INF] Active manager daemon smithi023.ehlpcl restarted 2021-11-19T08:35:42.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:42 smithi023 conmon[33490]: cluster 2021-11-19T08:35:42.545505+0000 mon.smithi023 (mon.0) 31 : cluster [INF] 2021-11-19T08:35:42.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:42 smithi023 conmon[33490]: Activating manager daemon smithi023.ehlpcl 2021-11-19T08:35:42.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:42 smithi023 conmon[33490]: cluster 2021-11-19T08:35:42.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:42 smithi023 conmon[33490]: 2021-11-19 2021-11-19T08:35:42.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:42 smithi023 conmon[33490]: T08:35:42.556318+0000 2021-11-19T08:35:42.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:42 smithi023 conmon[33490]: mon.smithi023 (mon.0) 32 : cluster [DBG] 2021-11-19T08:35:42.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:42 smithi023 conmon[33490]: osdmap e2: 0 total, 0 up, 0 in 2021-11-19T08:35:43.593 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.606+0000 7faa367fc700 1 -- 172.21.15.23:0/771401678 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7faa38010dc0 con 0x7faa4406fcd0 2021-11-19T08:35:43.593 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.607+0000 7faa367fc700 1 --2- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7faa2c033880 0x7faa2c035c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:43.594 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.607+0000 7faa367fc700 1 -- 172.21.15.23:0/771401678 --> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa2c032930 con 0x7faa2c033880 2021-11-19T08:35:43.595 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.607+0000 7faa4952a700 1 --2- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7faa2c033880 0x7faa2c035c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:43.595 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.608+0000 7faa4952a700 1 --2- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7faa2c033880 0x7faa2c035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7faa400036d0 tx=0x7faa40006c50).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:43.595 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.609+0000 7faa367fc700 1 -- 172.21.15.23:0/771401678 <== mgr.14116 v2:172.21.15.23:6800/1352671878 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7faa2c032930 con 0x7faa2c033880 2021-11-19T08:35:43.607 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.621+0000 7faa4bf8f700 1 -- 172.21.15.23:0/771401678 --> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7faa44059100 con 0x7faa2c033880 2021-11-19T08:35:43.607 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.621+0000 7faa367fc700 1 -- 172.21.15.23:0/771401678 <== mgr.14116 v2:172.21.15.23:6800/1352671878 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7faa44059100 con 0x7faa2c033880 2021-11-19T08:35:43.609 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2021-11-19T08:35:43.609 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2021-11-19T08:35:43.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "initialized": true 2021-11-19T08:35:43.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2021-11-19T08:35:43.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.621+0000 7faa4bf8f700 1 -- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7faa2c033880 msgr2=0x7faa2c035c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:43.611 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.621+0000 7faa4bf8f700 1 --2- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7faa2c033880 0x7faa2c035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7faa400036d0 tx=0x7faa40006c50).stop 2021-11-19T08:35:43.611 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.621+0000 7faa4bf8f700 1 -- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa4406fcd0 msgr2=0x7faa440fc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:43.611 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.621+0000 7faa4bf8f700 1 --2- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa4406fcd0 0x7faa440fc220 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7faa38000c00 tx=0x7faa38006ee0).stop 2021-11-19T08:35:43.611 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.621+0000 7faa4bf8f700 1 -- 172.21.15.23:0/771401678 shutdown_connections 2021-11-19T08:35:43.611 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.622+0000 7faa4bf8f700 1 --2- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7faa2c033880 0x7faa2c035c70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:43.612 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.622+0000 7faa4bf8f700 1 --2- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa4406fcd0 0x7faa440fc220 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:43.612 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.622+0000 7faa4bf8f700 1 --2- 172.21.15.23:0/771401678 >> [v2:172.21.15.23:6800/945698991,v1:172.21.15.23:6801/945698991] conn(0x7faa2c02fdb0 0x7faa2c032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:35:43.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.622+0000 7faa4bf8f700 1 -- 172.21.15.23:0/771401678 >> 172.21.15.23:0/771401678 conn(0x7faa440f6280 msgr2=0x7faa440f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:43.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.622+0000 7faa4bf8f700 1 -- 172.21.15.23:0/771401678 shutdown_connections 2021-11-19T08:35:43.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:43.622+0000 7faa4bf8f700 1 -- 172.21.15.23:0/771401678 wait complete. 2021-11-19T08:35:43.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: cluster 2021-11-19T08:35:42.600473+0000 mon.smithi023 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi023.ehlpcl(active, starting, since 0.0550894s) 2021-11-19T08:35:43.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit 2021-11-19T08:35:42.602104+0000 mon.smithi023 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi023"}]: dispatch 2021-11-19T08:35:43.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:35:43.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 11-19T08:35:42.602781 2021-11-19T08:35:43.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 35 2021-11-19T08:35:43.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: : audit [DBG] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mgr metadata", "who": "smithi023.ehlpcl", "id": "smithi023.ehlpcl"}]: dispatch 2021-11-19T08:35:43.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:35:43.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 11-19T08:35:42.604902 2021-11-19T08:35:43.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 36 : audit 2021-11-19T08:35:43.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: [DBG] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-19T08:35:43.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:35:43.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 11-19T08:35:42 2021-11-19T08:35:43.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: .605075+0000 mon.smithi023 (mon.0 2021-11-19T08:35:43.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: ) 37 : audit [DBG] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-19T08:35:43.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:35:43.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 11-19T08:35:42 2021-11-19T08:35:43.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: .605201+0000 mon.smithi023 (mon.0 2021-11-19T08:35:43.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: ) 38 : audit [DBG] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-19T08:35:43.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: cluster 2021-11 2021-11-19T08:35:43.898 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: -19T08:35:42.619450+0000 2021-11-19T08:35:43.898 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: mon.smithi023 (mon.0) 39 : cluster [INF] Manager daemon smithi023.ehlpcl is now available 2021-11-19T08:35:43.898 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 2021-11-19T08:35:43.898 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:35:43.899 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: T08:35:42.635449+0000 2021-11-19T08:35:43.899 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: mon.smithi023 (mon.0) 40 : audit [INF] 2021-11-19T08:35:43.899 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:43.899 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit 2021-11- 2021-11-19T08:35:43.899 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 19T08:35:42.638447 2021-11-19T08:35:43.900 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 41 : audit 2021-11-19T08:35:43.900 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:43.900 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit 2021-11- 2021-11-19T08:35:43.900 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 19T08:35:42. 2021-11-19T08:35:43.901 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 641265+0000 mon.smithi023 (mon.0) 42 : 2021-11-19T08:35:43.901 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:43.901 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit 2021-11-19T08:35:43.901 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 2021-11-19T08:35 2021-11-19T08:35:43.901 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: :42.765291+0000 mon.smithi023 (mon.0) 2021-11-19T08:35:43.902 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 43 : audit [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:43.902 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:35:43.902 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: -19T08:35:42. 2021-11-19T08:35:43.902 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 765991+0000 mon.smithi023 (mon.0) 44 : audit 2021-11-19T08:35:43.903 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: [DBG] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:35:43.903 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:35:43.903 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: -19T08:35:42. 2021-11-19T08:35:43.903 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 767471+0000 mon.smithi023 (mon.0) 45 : 2021-11-19T08:35:43.903 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit [DBG] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:35:43.904 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit 2021 2021-11-19T08:35:43.904 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: -11-19T08:35:42.769566+0000 2021-11-19T08:35:43.904 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: mon.smithi023 (mon.0) 46 : audit [DBG] 2021-11-19T08:35:43.904 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:35:43.905 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit 2021-11-19T08:35:43.905 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 2021-11-19T08:35 2021-11-19T08:35:43.905 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: :42.780940+0000 mon.smithi023 (mon.0 2021-11-19T08:35:43.905 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: ) 47 : audit [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:35:43.905 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: audit 2021-11-19T08:35:43.906 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 2021-11-19T08: 2021-11-19T08:35:43.906 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: 35:42.782238+0000 mon.smithi023 (mon 2021-11-19T08:35:43.906 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:43 smithi023 conmon[33490]: .0) 48 : audit [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:35:43.949 INFO:teuthology.orchestra.run.smithi023.stderr:mgr epoch 5 is available 2021-11-19T08:35:43.949 INFO:teuthology.orchestra.run.smithi023.stderr:Setting orchestrator backend to cephadm... 2021-11-19T08:35:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2021-11-19T08:35:44.423 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.435+0000 7fb6e0ef3700 1 Processor -- start 2021-11-19T08:35:44.423 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.435+0000 7fb6e0ef3700 1 -- start start 2021-11-19T08:35:44.423 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.436+0000 7fb6e0ef3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6dc0fd090 0x7fb6dc0fd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:44.424 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.436+0000 7fb6e0ef3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc0fd9a0 con 0x7fb6dc0fd090 2021-11-19T08:35:44.424 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.436+0000 7fb6db7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6dc0fd090 0x7fb6dc0fd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:44.424 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.436+0000 7fb6db7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6dc0fd090 0x7fb6dc0fd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46036/0 (socket says 172.21.15.23:46036) 2021-11-19T08:35:44.424 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.436+0000 7fb6db7fe700 1 -- 172.21.15.23:0/3964733210 learned_addr learned my addr 172.21.15.23:0/3964733210 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:44.424 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.436+0000 7fb6db7fe700 1 -- 172.21.15.23:0/3964733210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc0fdae0 con 0x7fb6dc0fd090 2021-11-19T08:35:44.425 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.436+0000 7fb6db7fe700 1 --2- 172.21.15.23:0/3964733210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6dc0fd090 0x7fb6dc0fd4b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb6cc00b5f0 tx=0x7fb6cc006ee0).ready entity=mon.0 client_cookie=3fa63b6ef677bfac server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:44.425 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.436+0000 7fb6da7fc700 1 -- 172.21.15.23:0/3964733210 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6cc00d600 con 0x7fb6dc0fd090 2021-11-19T08:35:44.427 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.436+0000 7fb6da7fc700 1 -- 172.21.15.23:0/3964733210 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb6cc00dbd0 con 0x7fb6dc0fd090 2021-11-19T08:35:44.427 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.436+0000 7fb6da7fc700 1 -- 172.21.15.23:0/3964733210 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6cc011c00 con 0x7fb6dc0fd090 2021-11-19T08:35:44.427 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.437+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3964733210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6dc0fd090 msgr2=0x7fb6dc0fd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:44.428 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.437+0000 7fb6e0ef3700 1 --2- 172.21.15.23:0/3964733210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6dc0fd090 0x7fb6dc0fd4b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb6cc00b5f0 tx=0x7fb6cc006ee0).stop 2021-11-19T08:35:44.428 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.437+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3964733210 shutdown_connections 2021-11-19T08:35:44.428 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.437+0000 7fb6e0ef3700 1 --2- 172.21.15.23:0/3964733210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6dc0fd090 0x7fb6dc0fd4b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:44.428 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.437+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3964733210 >> 172.21.15.23:0/3964733210 conn(0x7fb6dc0f5870 msgr2=0x7fb6dc0f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:44.428 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.437+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3964733210 shutdown_connections 2021-11-19T08:35:44.429 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.437+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3964733210 wait complete. 2021-11-19T08:35:44.429 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.438+0000 7fb6e0ef3700 1 Processor -- start 2021-11-19T08:35:44.429 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.438+0000 7fb6e0ef3700 1 -- start start 2021-11-19T08:35:44.429 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.438+0000 7fb6e0ef3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6dc0fd090 0x7fb6dc0fc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:44.430 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.438+0000 7fb6e0ef3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc0fade0 con 0x7fb6dc0fd090 2021-11-19T08:35:44.430 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.438+0000 7fb6db7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6dc0fd090 0x7fb6dc0fc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:44.430 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.438+0000 7fb6db7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6dc0fd090 0x7fb6dc0fc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46038/0 (socket says 172.21.15.23:46038) 2021-11-19T08:35:44.430 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.438+0000 7fb6db7fe700 1 -- 172.21.15.23:0/3156185019 learned_addr learned my addr 172.21.15.23:0/3156185019 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:44.430 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.439+0000 7fb6db7fe700 1 -- 172.21.15.23:0/3156185019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc00b200 con 0x7fb6dc0fd090 2021-11-19T08:35:44.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.439+0000 7fb6db7fe700 1 --2- 172.21.15.23:0/3156185019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6dc0fd090 0x7fb6dc0fc960 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fb6cc000c00 tx=0x7fb6cc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:44.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.439+0000 7fb6c3fff700 1 -- 172.21.15.23:0/3156185019 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6cc011690 con 0x7fb6dc0fd090 2021-11-19T08:35:44.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.439+0000 7fb6c3fff700 1 -- 172.21.15.23:0/3156185019 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb6cc00d9f0 con 0x7fb6dc0fd090 2021-11-19T08:35:44.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.439+0000 7fb6c3fff700 1 -- 172.21.15.23:0/3156185019 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6cc01b710 con 0x7fb6dc0fd090 2021-11-19T08:35:44.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.439+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3156185019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc0faf80 con 0x7fb6dc0fd090 2021-11-19T08:35:44.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.439+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3156185019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc0fb390 con 0x7fb6dc0fd090 2021-11-19T08:35:44.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.440+0000 7fb6c3fff700 1 -- 172.21.15.23:0/3156185019 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fb6cc00f070 con 0x7fb6dc0fd090 2021-11-19T08:35:44.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.440+0000 7fb6c3fff700 1 --2- 172.21.15.23:0/3156185019 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fb6c802fca0 0x7fb6c8032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:44.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.440+0000 7fb6c3fff700 1 -- 172.21.15.23:0/3156185019 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb6cc0410a0 con 0x7fb6dc0fd090 2021-11-19T08:35:44.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.440+0000 7fb6daffd700 1 --2- 172.21.15.23:0/3156185019 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fb6c802fca0 0x7fb6c8032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:44.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.440+0000 7fb6daffd700 1 --2- 172.21.15.23:0/3156185019 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fb6c802fca0 0x7fb6c8032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb6d00096a0 tx=0x7fb6d0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:44.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.441+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3156185019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6dc059100 con 0x7fb6dc0fd090 2021-11-19T08:35:44.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.445+0000 7fb6c3fff700 1 -- 172.21.15.23:0/3156185019 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6cc016070 con 0x7fb6dc0fd090 2021-11-19T08:35:44.776 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:44 smithi023 conmon[33490]: audit 2021-11-19T08:35:43.609522+0000 mgr.smithi023.ehlpcl (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-11-19T08:35:44.776 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:44 smithi023 conmon[33490]: cluster 2021-11-19T08:35:43.610504+0000 mon.smithi023 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi023.ehlpcl(active, since 1.06512s) 2021-11-19T08:35:44.777 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:44 smithi023 conmon[33490]: audit 2021-11-19T08:35:43.622159+0000 mgr.smithi023.ehlpcl (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-11-19T08:35:44.777 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.790+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3156185019 --> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6dc0fa4d0 con 0x7fb6c802fca0 2021-11-19T08:35:44.783 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.798+0000 7fb6c3fff700 1 -- 172.21.15.23:0/3156185019 <== mgr.14116 v2:172.21.15.23:6800/1352671878 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb6dc0fa4d0 con 0x7fb6c802fca0 2021-11-19T08:35:44.786 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.799+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3156185019 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fb6c802fca0 msgr2=0x7fb6c8032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:44.786 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.799+0000 7fb6e0ef3700 1 --2- 172.21.15.23:0/3156185019 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fb6c802fca0 0x7fb6c8032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb6d00096a0 tx=0x7fb6d0006b40).stop 2021-11-19T08:35:44.787 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.799+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3156185019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6dc0fd090 msgr2=0x7fb6dc0fc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:44.787 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.799+0000 7fb6e0ef3700 1 --2- 172.21.15.23:0/3156185019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6dc0fd090 0x7fb6dc0fc960 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fb6cc000c00 tx=0x7fb6cc006ee0).stop 2021-11-19T08:35:44.787 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.799+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3156185019 shutdown_connections 2021-11-19T08:35:44.787 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.800+0000 7fb6e0ef3700 1 --2- 172.21.15.23:0/3156185019 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fb6c802fca0 0x7fb6c8032160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:44.788 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.800+0000 7fb6e0ef3700 1 --2- 172.21.15.23:0/3156185019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6dc0fd090 0x7fb6dc0fc960 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:44.788 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.800+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3156185019 >> 172.21.15.23:0/3156185019 conn(0x7fb6dc0f5870 msgr2=0x7fb6dc0ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:44.788 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.800+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3156185019 shutdown_connections 2021-11-19T08:35:44.788 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:44.800+0000 7fb6e0ef3700 1 -- 172.21.15.23:0/3156185019 wait complete. 2021-11-19T08:35:45.187 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2021-11-19T08:35:45.733 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.743+0000 7f92f7777700 1 Processor -- start 2021-11-19T08:35:45.734 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.743+0000 7f92f7777700 1 -- start start 2021-11-19T08:35:45.734 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.743+0000 7f92f7777700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92f00fc210 0x7f92f00fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:45.734 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.743+0000 7f92f7777700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f00fcb20 con 0x7f92f00fc210 2021-11-19T08:35:45.735 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.743+0000 7f92f5513700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92f00fc210 0x7f92f00fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:45.735 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.744+0000 7f92f5513700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92f00fc210 0x7f92f00fc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46042/0 (socket says 172.21.15.23:46042) 2021-11-19T08:35:45.735 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.744+0000 7f92f5513700 1 -- 172.21.15.23:0/705143191 learned_addr learned my addr 172.21.15.23:0/705143191 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:45.735 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.744+0000 7f92f5513700 1 -- 172.21.15.23:0/705143191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f00fcc60 con 0x7f92f00fc210 2021-11-19T08:35:45.736 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.744+0000 7f92f5513700 1 --2- 172.21.15.23:0/705143191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92f00fc210 0x7f92f00fc630 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f92ec00b5f0 tx=0x7f92ec006ee0).ready entity=mon.0 client_cookie=7e0522ed8e5dac4f server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:45.736 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.744+0000 7f92dffff700 1 -- 172.21.15.23:0/705143191 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92ec00d600 con 0x7f92f00fc210 2021-11-19T08:35:45.736 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.744+0000 7f92dffff700 1 -- 172.21.15.23:0/705143191 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f92ec00dbd0 con 0x7f92f00fc210 2021-11-19T08:35:45.736 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.744+0000 7f92dffff700 1 -- 172.21.15.23:0/705143191 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92ec011c00 con 0x7f92f00fc210 2021-11-19T08:35:45.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.745+0000 7f92f7777700 1 -- 172.21.15.23:0/705143191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92f00fc210 msgr2=0x7f92f00fc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:45.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.745+0000 7f92f7777700 1 --2- 172.21.15.23:0/705143191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92f00fc210 0x7f92f00fc630 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f92ec00b5f0 tx=0x7f92ec006ee0).stop 2021-11-19T08:35:45.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.745+0000 7f92f7777700 1 -- 172.21.15.23:0/705143191 shutdown_connections 2021-11-19T08:35:45.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.745+0000 7f92f7777700 1 --2- 172.21.15.23:0/705143191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92f00fc210 0x7f92f00fc630 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:45.738 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.745+0000 7f92f7777700 1 -- 172.21.15.23:0/705143191 >> 172.21.15.23:0/705143191 conn(0x7f92f00f5a80 msgr2=0x7f92f00f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:45.738 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.745+0000 7f92f7777700 1 -- 172.21.15.23:0/705143191 shutdown_connections 2021-11-19T08:35:45.738 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.745+0000 7f92f7777700 1 -- 172.21.15.23:0/705143191 wait complete. 2021-11-19T08:35:45.738 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.745+0000 7f92f7777700 1 Processor -- start 2021-11-19T08:35:45.739 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.746+0000 7f92f7777700 1 -- start start 2021-11-19T08:35:45.739 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.746+0000 7f92f7777700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92f00fc210 0x7f92f010cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:45.739 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.746+0000 7f92f7777700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f010d0f0 con 0x7f92f00fc210 2021-11-19T08:35:45.739 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.746+0000 7f92f5513700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92f00fc210 0x7f92f010cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:45.740 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.746+0000 7f92f5513700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92f00fc210 0x7f92f010cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46044/0 (socket says 172.21.15.23:46044) 2021-11-19T08:35:45.740 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.746+0000 7f92f5513700 1 -- 172.21.15.23:0/2263805222 learned_addr learned my addr 172.21.15.23:0/2263805222 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:45.740 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.746+0000 7f92f5513700 1 -- 172.21.15.23:0/2263805222 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ec00b200 con 0x7f92f00fc210 2021-11-19T08:35:45.741 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.746+0000 7f92f5513700 1 --2- 172.21.15.23:0/2263805222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92f00fc210 0x7f92f010cc00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f92ec000c00 tx=0x7f92ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:45.741 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.746+0000 7f92ddffb700 1 -- 172.21.15.23:0/2263805222 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92ec011690 con 0x7f92f00fc210 2021-11-19T08:35:45.741 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.746+0000 7f92f7777700 1 -- 172.21.15.23:0/2263805222 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f010d2f0 con 0x7f92f00fc210 2021-11-19T08:35:45.742 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.746+0000 7f92ddffb700 1 -- 172.21.15.23:0/2263805222 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f92ec00d9f0 con 0x7f92f00fc210 2021-11-19T08:35:45.742 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.746+0000 7f92ddffb700 1 -- 172.21.15.23:0/2263805222 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92ec01b710 con 0x7f92f00fc210 2021-11-19T08:35:45.742 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.747+0000 7f92f7777700 1 -- 172.21.15.23:0/2263805222 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f010d700 con 0x7f92f00fc210 2021-11-19T08:35:45.743 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.747+0000 7f92ddffb700 1 -- 172.21.15.23:0/2263805222 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f92ec00f070 con 0x7f92f00fc210 2021-11-19T08:35:45.743 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.747+0000 7f92ddffb700 1 --2- 172.21.15.23:0/2263805222 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f92d802fd50 0x7f92d8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:45.744 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.748+0000 7f92ddffb700 1 -- 172.21.15.23:0/2263805222 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f92ec041170 con 0x7f92f00fc210 2021-11-19T08:35:45.745 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.748+0000 7f92f4d12700 1 --2- 172.21.15.23:0/2263805222 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f92d802fd50 0x7f92d8032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:45.745 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.748+0000 7f92f4d12700 1 --2- 172.21.15.23:0/2263805222 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f92d802fd50 0x7f92d8032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f92e00096a0 tx=0x7f92e0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:45.746 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.748+0000 7f92f7777700 1 -- 172.21.15.23:0/2263805222 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f0059100 con 0x7f92f00fc210 2021-11-19T08:35:45.746 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:45.753+0000 7f92ddffb700 1 -- 172.21.15.23:0/2263805222 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92ec016070 con 0x7f92f00fc210 2021-11-19T08:35:46.031 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:45 smithi023 conmon[33490]: audit 2021-11-19T08:35:44.792277+0000 mgr.smithi023.ehlpcl 2021-11-19T08:35:46.032 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:45 smithi023 conmon[33490]: (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:35:46.032 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:45 smithi023 conmon[33490]: audit 2021-11-19T08:35:44.798450+0000 mon.smithi023 (mon. 2021-11-19T08:35:46.032 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:45 smithi023 conmon[33490]: 0) 50 : audit [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:46.032 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:45 smithi023 conmon[33490]: audit 2021 2021-11-19T08:35:46.033 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:45 smithi023 conmon[33490]: -11-19T08:35:44. 2021-11-19T08:35:46.033 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:45 smithi023 conmon[33490]: 802449+0000 mon.smithi023 (mon.0) 2021-11-19T08:35:46.033 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:45 smithi023 conmon[33490]: 51 : audit [DBG] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:35:46.033 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:45 smithi023 conmon[33490]: cluster 2021 2021-11-19T08:35:46.034 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:45 smithi023 conmon[33490]: -11-19T08:35: 2021-11-19T08:35:46.034 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:45 smithi023 conmon[33490]: 45.108625+0000 mon.smithi023 (mon. 2021-11-19T08:35:46.034 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:45 smithi023 conmon[33490]: 0) 52 : cluster [DBG] 2021-11-19T08:35:46.034 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:45 smithi023 conmon[33490]: mgrmap e8: smithi023.ehlpcl(active, since 2s) 2021-11-19T08:35:46.036 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.044+0000 7f92f7777700 1 -- 172.21.15.23:0/2263805222 --> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f92f01069f0 con 0x7f92d802fd50 2021-11-19T08:35:46.037 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.045+0000 7f92ddffb700 1 -- 172.21.15.23:0/2263805222 <== mgr.14116 v2:172.21.15.23:6800/1352671878 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f92f01069f0 con 0x7f92d802fd50 2021-11-19T08:35:46.037 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: value unchanged 2021-11-19T08:35:46.037 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.047+0000 7f92f7777700 1 -- 172.21.15.23:0/2263805222 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f92d802fd50 msgr2=0x7f92d8032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:46.038 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.047+0000 7f92f7777700 1 --2- 172.21.15.23:0/2263805222 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f92d802fd50 0x7f92d8032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f92e00096a0 tx=0x7f92e0006b40).stop 2021-11-19T08:35:46.038 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.047+0000 7f92f7777700 1 -- 172.21.15.23:0/2263805222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92f00fc210 msgr2=0x7f92f010cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:46.038 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.047+0000 7f92f7777700 1 --2- 172.21.15.23:0/2263805222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92f00fc210 0x7f92f010cc00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f92ec000c00 tx=0x7f92ec006ee0).stop 2021-11-19T08:35:46.038 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.047+0000 7f92f7777700 1 -- 172.21.15.23:0/2263805222 shutdown_connections 2021-11-19T08:35:46.039 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.047+0000 7f92f7777700 1 --2- 172.21.15.23:0/2263805222 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f92d802fd50 0x7f92d8032210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:46.039 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.047+0000 7f92f7777700 1 --2- 172.21.15.23:0/2263805222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92f00fc210 0x7f92f010cc00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:46.039 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.047+0000 7f92f7777700 1 -- 172.21.15.23:0/2263805222 >> 172.21.15.23:0/2263805222 conn(0x7f92f00f5a80 msgr2=0x7f92f00f7620 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:46.039 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.048+0000 7f92f7777700 1 -- 172.21.15.23:0/2263805222 shutdown_connections 2021-11-19T08:35:46.040 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.048+0000 7f92f7777700 1 -- 172.21.15.23:0/2263805222 wait complete. 2021-11-19T08:35:46.424 INFO:teuthology.orchestra.run.smithi023.stderr:Generating ssh key... 2021-11-19T08:35:46.424 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2021-11-19T08:35:46.911 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:46 smithi023 conmon[33490]: audit 2021-11-19T08:35:46.046225+0000 mgr.smithi023.ehlpcl 2021-11-19T08:35:46.912 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:46 smithi023 conmon[33490]: (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:35:46.915 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.925+0000 7f10b8cad700 1 Processor -- start 2021-11-19T08:35:46.916 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.925+0000 7f10b8cad700 1 -- start start 2021-11-19T08:35:46.916 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.925+0000 7f10b8cad700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10b40fc1d0 0x7f10b40fc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:46.916 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.926+0000 7f10b8cad700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b40fcae0 con 0x7f10b40fc1d0 2021-11-19T08:35:46.916 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.926+0000 7f10b259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10b40fc1d0 0x7f10b40fc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:46.917 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.926+0000 7f10b259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10b40fc1d0 0x7f10b40fc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46048/0 (socket says 172.21.15.23:46048) 2021-11-19T08:35:46.917 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.926+0000 7f10b259c700 1 -- 172.21.15.23:0/1519145560 learned_addr learned my addr 172.21.15.23:0/1519145560 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:46.917 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.926+0000 7f10b259c700 1 -- 172.21.15.23:0/1519145560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b40fcc20 con 0x7f10b40fc1d0 2021-11-19T08:35:46.917 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.926+0000 7f10b259c700 1 --2- 172.21.15.23:0/1519145560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10b40fc1d0 0x7f10b40fc5f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f10ac009960 tx=0x7f10ac0092d0).ready entity=mon.0 client_cookie=5159cb2d76ee5db2 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:46.917 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.926+0000 7f10b159a700 1 -- 172.21.15.23:0/1519145560 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10ac00b9c0 con 0x7f10b40fc1d0 2021-11-19T08:35:46.918 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.926+0000 7f10b159a700 1 -- 172.21.15.23:0/1519145560 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f10ac011400 con 0x7f10b40fc1d0 2021-11-19T08:35:46.918 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.926+0000 7f10b159a700 1 -- 172.21.15.23:0/1519145560 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10ac0103a0 con 0x7f10b40fc1d0 2021-11-19T08:35:46.918 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.927+0000 7f10b8cad700 1 -- 172.21.15.23:0/1519145560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10b40fc1d0 msgr2=0x7f10b40fc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:46.918 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.927+0000 7f10b8cad700 1 --2- 172.21.15.23:0/1519145560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10b40fc1d0 0x7f10b40fc5f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f10ac009960 tx=0x7f10ac0092d0).stop 2021-11-19T08:35:46.919 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.927+0000 7f10b8cad700 1 -- 172.21.15.23:0/1519145560 shutdown_connections 2021-11-19T08:35:46.919 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.927+0000 7f10b8cad700 1 --2- 172.21.15.23:0/1519145560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10b40fc1d0 0x7f10b40fc5f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:46.919 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.927+0000 7f10b8cad700 1 -- 172.21.15.23:0/1519145560 >> 172.21.15.23:0/1519145560 conn(0x7f10b40f5a80 msgr2=0x7f10b40f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:46.919 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.927+0000 7f10b8cad700 1 -- 172.21.15.23:0/1519145560 shutdown_connections 2021-11-19T08:35:46.919 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.927+0000 7f10b8cad700 1 -- 172.21.15.23:0/1519145560 wait complete. 2021-11-19T08:35:46.920 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.928+0000 7f10b8cad700 1 Processor -- start 2021-11-19T08:35:46.920 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.928+0000 7f10b8cad700 1 -- start start 2021-11-19T08:35:46.920 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.928+0000 7f10b8cad700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10b40fc1d0 0x7f10b410cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:46.920 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.928+0000 7f10b8cad700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b410d0a0 con 0x7f10b40fc1d0 2021-11-19T08:35:46.921 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.928+0000 7f10b259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10b40fc1d0 0x7f10b410cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:46.921 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.928+0000 7f10b259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10b40fc1d0 0x7f10b410cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46050/0 (socket says 172.21.15.23:46050) 2021-11-19T08:35:46.921 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.928+0000 7f10b259c700 1 -- 172.21.15.23:0/3799803605 learned_addr learned my addr 172.21.15.23:0/3799803605 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:46.921 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.928+0000 7f10b259c700 1 -- 172.21.15.23:0/3799803605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ac009570 con 0x7f10b40fc1d0 2021-11-19T08:35:46.922 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.929+0000 7f10b259c700 1 --2- 172.21.15.23:0/3799803605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10b40fc1d0 0x7f10b410cbb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f10ac00bac0 tx=0x7f10ac0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:46.922 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.929+0000 7f10aaffd700 1 -- 172.21.15.23:0/3799803605 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10ac010870 con 0x7f10b40fc1d0 2021-11-19T08:35:46.922 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.929+0000 7f10b8cad700 1 -- 172.21.15.23:0/3799803605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b410d2a0 con 0x7f10b40fc1d0 2021-11-19T08:35:46.922 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.929+0000 7f10b8cad700 1 -- 172.21.15.23:0/3799803605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b410d6b0 con 0x7f10b40fc1d0 2021-11-19T08:35:46.922 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.929+0000 7f10aaffd700 1 -- 172.21.15.23:0/3799803605 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f10ac010e40 con 0x7f10b40fc1d0 2021-11-19T08:35:46.923 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.929+0000 7f10aaffd700 1 -- 172.21.15.23:0/3799803605 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10ac01ae00 con 0x7f10b40fc1d0 2021-11-19T08:35:46.923 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.930+0000 7f10aaffd700 1 -- 172.21.15.23:0/3799803605 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f10ac00f070 con 0x7f10b40fc1d0 2021-11-19T08:35:46.924 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.930+0000 7f10aaffd700 1 --2- 172.21.15.23:0/3799803605 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f109402fd50 0x7f1094032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:46.924 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.930+0000 7f10b8cad700 1 -- 172.21.15.23:0/3799803605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b4045ad0 con 0x7f10b40fc1d0 2021-11-19T08:35:46.924 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.930+0000 7f10aaffd700 1 -- 172.21.15.23:0/3799803605 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f10ac041f30 con 0x7f10b40fc1d0 2021-11-19T08:35:46.925 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.930+0000 7f10b1d9b700 1 --2- 172.21.15.23:0/3799803605 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f109402fd50 0x7f1094032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:46.925 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.930+0000 7f10b1d9b700 1 --2- 172.21.15.23:0/3799803605 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f109402fd50 0x7f1094032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f109c0096a0 tx=0x7f109c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:46.925 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:46.934+0000 7f10aaffd700 1 -- 172.21.15.23:0/3799803605 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10ac01e030 con 0x7f10b40fc1d0 2021-11-19T08:35:47.209 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:47.223+0000 7f10b8cad700 1 -- 172.21.15.23:0/3799803605 --> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f10b410df90 con 0x7f109402fd50 2021-11-19T08:35:47.458 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:47.472+0000 7f10aaffd700 1 -- 172.21.15.23:0/3799803605 <== mgr.14116 v2:172.21.15.23:6800/1352671878 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f10b410df90 con 0x7f109402fd50 2021-11-19T08:35:47.461 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:47.474+0000 7f10b8cad700 1 -- 172.21.15.23:0/3799803605 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f109402fd50 msgr2=0x7f1094032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:47.461 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:47.474+0000 7f10b8cad700 1 --2- 172.21.15.23:0/3799803605 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f109402fd50 0x7f1094032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f109c0096a0 tx=0x7f109c006b40).stop 2021-11-19T08:35:47.462 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:47.474+0000 7f10b8cad700 1 -- 172.21.15.23:0/3799803605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10b40fc1d0 msgr2=0x7f10b410cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:47.462 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:47.474+0000 7f10b8cad700 1 --2- 172.21.15.23:0/3799803605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10b40fc1d0 0x7f10b410cbb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f10ac00bac0 tx=0x7f10ac0092d0).stop 2021-11-19T08:35:47.462 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:47.474+0000 7f10b8cad700 1 -- 172.21.15.23:0/3799803605 shutdown_connections 2021-11-19T08:35:47.462 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:47.474+0000 7f10b8cad700 1 --2- 172.21.15.23:0/3799803605 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f109402fd50 0x7f1094032210 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:47.463 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:47.474+0000 7f10b8cad700 1 --2- 172.21.15.23:0/3799803605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10b40fc1d0 0x7f10b410cbb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:47.463 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:47.474+0000 7f10b8cad700 1 -- 172.21.15.23:0/3799803605 >> 172.21.15.23:0/3799803605 conn(0x7f10b40f5a80 msgr2=0x7f10b40f75d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:47.463 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:47.474+0000 7f10b8cad700 1 -- 172.21.15.23:0/3799803605 shutdown_connections 2021-11-19T08:35:47.463 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:47.475+0000 7f10b8cad700 1 -- 172.21.15.23:0/3799803605 wait complete. 2021-11-19T08:35:47.860 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2021-11-19T08:35:48.315 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.325+0000 7fa80e8f0700 1 Processor -- start 2021-11-19T08:35:48.315 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.325+0000 7fa80e8f0700 1 -- start start 2021-11-19T08:35:48.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.325+0000 7fa80e8f0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa808071c90 0x7fa808070230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:48.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.325+0000 7fa80e8f0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8080720b0 con 0x7fa808071c90 2021-11-19T08:35:48.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.326+0000 7fa80d8ee700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa808071c90 0x7fa808070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:48.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.326+0000 7fa80d8ee700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa808071c90 0x7fa808070230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46054/0 (socket says 172.21.15.23:46054) 2021-11-19T08:35:48.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.326+0000 7fa80d8ee700 1 -- 172.21.15.23:0/1606465270 learned_addr learned my addr 172.21.15.23:0/1606465270 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:48.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.326+0000 7fa80d8ee700 1 -- 172.21.15.23:0/1606465270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa808070720 con 0x7fa808071c90 2021-11-19T08:35:48.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.326+0000 7fa80d8ee700 1 --2- 172.21.15.23:0/1606465270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa808071c90 0x7fa808070230 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fa800009960 tx=0x7fa8000092d0).ready entity=mon.0 client_cookie=17609a49b0e0d106 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:48.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.326+0000 7fa80c8ec700 1 -- 172.21.15.23:0/1606465270 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa80000b9c0 con 0x7fa808071c90 2021-11-19T08:35:48.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.326+0000 7fa80c8ec700 1 -- 172.21.15.23:0/1606465270 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa800011400 con 0x7fa808071c90 2021-11-19T08:35:48.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.326+0000 7fa80c8ec700 1 -- 172.21.15.23:0/1606465270 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8000103a0 con 0x7fa808071c90 2021-11-19T08:35:48.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.327+0000 7fa80e8f0700 1 -- 172.21.15.23:0/1606465270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa808071c90 msgr2=0x7fa808070230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:48.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.327+0000 7fa80e8f0700 1 --2- 172.21.15.23:0/1606465270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa808071c90 0x7fa808070230 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fa800009960 tx=0x7fa8000092d0).stop 2021-11-19T08:35:48.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.327+0000 7fa80e8f0700 1 -- 172.21.15.23:0/1606465270 shutdown_connections 2021-11-19T08:35:48.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.327+0000 7fa80e8f0700 1 --2- 172.21.15.23:0/1606465270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa808071c90 0x7fa808070230 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:48.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.327+0000 7fa80e8f0700 1 -- 172.21.15.23:0/1606465270 >> 172.21.15.23:0/1606465270 conn(0x7fa8080f71e0 msgr2=0x7fa8080f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:48.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.327+0000 7fa80e8f0700 1 -- 172.21.15.23:0/1606465270 shutdown_connections 2021-11-19T08:35:48.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.327+0000 7fa80e8f0700 1 -- 172.21.15.23:0/1606465270 wait complete. 2021-11-19T08:35:48.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.327+0000 7fa80e8f0700 1 Processor -- start 2021-11-19T08:35:48.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.328+0000 7fa80e8f0700 1 -- start start 2021-11-19T08:35:48.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.328+0000 7fa80e8f0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa808071c90 0x7fa80810c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:48.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.328+0000 7fa80e8f0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80810ce90 con 0x7fa808071c90 2021-11-19T08:35:48.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.328+0000 7fa80d8ee700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa808071c90 0x7fa80810c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:48.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.328+0000 7fa80d8ee700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa808071c90 0x7fa80810c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46056/0 (socket says 172.21.15.23:46056) 2021-11-19T08:35:48.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.328+0000 7fa80d8ee700 1 -- 172.21.15.23:0/2506379431 learned_addr learned my addr 172.21.15.23:0/2506379431 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:48.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.328+0000 7fa80d8ee700 1 -- 172.21.15.23:0/2506379431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa800009570 con 0x7fa808071c90 2021-11-19T08:35:48.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.328+0000 7fa80d8ee700 1 --2- 172.21.15.23:0/2506379431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa808071c90 0x7fa80810c9a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa800006b80 tx=0x7fa8000092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:48.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.328+0000 7fa8067fc700 1 -- 172.21.15.23:0/2506379431 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa800010870 con 0x7fa808071c90 2021-11-19T08:35:48.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.328+0000 7fa8067fc700 1 -- 172.21.15.23:0/2506379431 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa800010e40 con 0x7fa808071c90 2021-11-19T08:35:48.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.328+0000 7fa80e8f0700 1 -- 172.21.15.23:0/2506379431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80810d090 con 0x7fa808071c90 2021-11-19T08:35:48.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.329+0000 7fa8067fc700 1 -- 172.21.15.23:0/2506379431 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa80001ad60 con 0x7fa808071c90 2021-11-19T08:35:48.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.329+0000 7fa80e8f0700 1 -- 172.21.15.23:0/2506379431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa80810d4a0 con 0x7fa808071c90 2021-11-19T08:35:48.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.329+0000 7fa8067fc700 1 -- 172.21.15.23:0/2506379431 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa80002a070 con 0x7fa808071c90 2021-11-19T08:35:48.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.330+0000 7fa8067fc700 1 --2- 172.21.15.23:0/2506379431 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fa7f402fdb0 0x7fa7f4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:48.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.330+0000 7fa8067fc700 1 -- 172.21.15.23:0/2506379431 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa800041a80 con 0x7fa808071c90 2021-11-19T08:35:48.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.330+0000 7fa80d0ed700 1 --2- 172.21.15.23:0/2506379431 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fa7f402fdb0 0x7fa7f4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:48.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.330+0000 7fa80e8f0700 1 -- 172.21.15.23:0/2506379431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa808045ad0 con 0x7fa808071c90 2021-11-19T08:35:48.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.330+0000 7fa80d0ed700 1 --2- 172.21.15.23:0/2506379431 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fa7f402fdb0 0x7fa7f4032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa7fc0096a0 tx=0x7fa7fc006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:48.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.334+0000 7fa8067fc700 1 -- 172.21.15.23:0/2506379431 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa80001e020 con 0x7fa808071c90 2021-11-19T08:35:48.609 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.623+0000 7fa80e8f0700 1 -- 172.21.15.23:0/2506379431 --> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa808106840 con 0x7fa7f402fdb0 2021-11-19T08:35:48.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.623+0000 7fa8067fc700 1 -- 172.21.15.23:0/2506379431 <== mgr.14116 v2:172.21.15.23:6800/1352671878 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fa808106840 con 0x7fa7f402fdb0 2021-11-19T08:35:48.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm+nRJhuzu3Xo9dgX2/W0ZvfH7Pv7PZRW+AIKnun+byC+NyRAiAGA8q4IYKAim7Yj7obvHzxhY8QVBxEJP/1l3ogYzbqrdOQNiX6dAKTSUF1FsbdkXa9ktPlqkj+MM0962nEhsqtpMm5kmzoqUTTuoGYlL3KFMvLLzaP8PzPuZORYcJD/KBCZcRwPDiawFtj9nnof2/7T2nLFhXlP5bhPc2Gfco40k17EQMa4Sp2Cm3U2TYdGk+K/iOnH1uMWUi5K7fJ+y5W+nMpOu9ng2gtl3kAV+Cr0XBGrv2BkfFrboxddrv19KKEMxmmBI70cBteBxGfPexpcUk8A9z9UO3IrWidAtf1CTgbK+3h0Y1zFf4I17xT3iyv6yl2GuyaWYkHm9lQ3Aa7t/vAxQGmkOGj6H9NUYAzxk+z1htKGIOagjkuLhoTeb65haQDcjw8gxTjpqNv3u3zxEIVOzLL0cYfcUMP+lJc5oumfCowI2jdwqPKEh5cqGMRHdHQmXxjtgTT0= ceph-908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:35:48.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.625+0000 7fa80e8f0700 1 -- 172.21.15.23:0/2506379431 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fa7f402fdb0 msgr2=0x7fa7f4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:48.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.625+0000 7fa80e8f0700 1 --2- 172.21.15.23:0/2506379431 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fa7f402fdb0 0x7fa7f4032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa7fc0096a0 tx=0x7fa7fc006b40).stop 2021-11-19T08:35:48.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.625+0000 7fa80e8f0700 1 -- 172.21.15.23:0/2506379431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa808071c90 msgr2=0x7fa80810c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:48.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.625+0000 7fa80e8f0700 1 --2- 172.21.15.23:0/2506379431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa808071c90 0x7fa80810c9a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa800006b80 tx=0x7fa8000092d0).stop 2021-11-19T08:35:48.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.626+0000 7fa80e8f0700 1 -- 172.21.15.23:0/2506379431 shutdown_connections 2021-11-19T08:35:48.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.626+0000 7fa80e8f0700 1 --2- 172.21.15.23:0/2506379431 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fa7f402fdb0 0x7fa7f4032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:48.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.626+0000 7fa80e8f0700 1 --2- 172.21.15.23:0/2506379431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa808071c90 0x7fa80810c9a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:48.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.626+0000 7fa80e8f0700 1 -- 172.21.15.23:0/2506379431 >> 172.21.15.23:0/2506379431 conn(0x7fa8080f71e0 msgr2=0x7fa8080f8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:48.615 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.626+0000 7fa80e8f0700 1 -- 172.21.15.23:0/2506379431 shutdown_connections 2021-11-19T08:35:48.615 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:48.626+0000 7fa80e8f0700 1 -- 172.21.15.23:0/2506379431 wait complete. 2021-11-19T08:35:48.787 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:48 smithi023 conmon[33490]: cephadm 2021-11-19T08:35:47.224786+0000 mgr.smithi023.ehlpcl (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2021-11-19T08:35:48.788 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:48 smithi023 conmon[33490]: audit 2021-11-19T08:35:47.468568+0000 mon.smithi023 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:48.788 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:48 smithi023 conmon[33490]: audit 2021-11-19T08 2021-11-19T08:35:48.788 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:48 smithi023 conmon[33490]: :35:47.472673+0000 mon.smithi023 (mon.0) 54 : 2021-11-19T08:35:48.788 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:48 smithi023 conmon[33490]: audit [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:48.996 INFO:teuthology.orchestra.run.smithi023.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2021-11-19T08:35:48.996 INFO:teuthology.orchestra.run.smithi023.stderr:Adding key to root@localhost authorized_keys... 2021-11-19T08:35:48.997 INFO:teuthology.orchestra.run.smithi023.stderr:Adding host smithi023... 2021-11-19T08:35:48.998 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi023 172.21.15.23 2021-11-19T08:35:49.467 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.475+0000 7fd5f1a7b700 1 Processor -- start 2021-11-19T08:35:49.467 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.475+0000 7fd5f1a7b700 1 -- start start 2021-11-19T08:35:49.468 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.475+0000 7fd5f1a7b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5ec0faad0 0x7fd5ec0faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:49.468 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.475+0000 7fd5f1a7b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec0fb3e0 con 0x7fd5ec0faad0 2021-11-19T08:35:49.468 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.475+0000 7fd5eb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5ec0faad0 0x7fd5ec0faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:49.468 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.475+0000 7fd5eb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5ec0faad0 0x7fd5ec0faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46060/0 (socket says 172.21.15.23:46060) 2021-11-19T08:35:49.469 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.475+0000 7fd5eb7fe700 1 -- 172.21.15.23:0/3424842393 learned_addr learned my addr 172.21.15.23:0/3424842393 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:49.469 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.476+0000 7fd5eb7fe700 1 -- 172.21.15.23:0/3424842393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec0fb520 con 0x7fd5ec0faad0 2021-11-19T08:35:49.469 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.476+0000 7fd5eb7fe700 1 --2- 172.21.15.23:0/3424842393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5ec0faad0 0x7fd5ec0faef0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fd5dc009960 tx=0x7fd5dc0092d0).ready entity=mon.0 client_cookie=2256870b341e50b3 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:49.470 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.476+0000 7fd5ea7fc700 1 -- 172.21.15.23:0/3424842393 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5dc00b9c0 con 0x7fd5ec0faad0 2021-11-19T08:35:49.470 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.476+0000 7fd5ea7fc700 1 -- 172.21.15.23:0/3424842393 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd5dc011400 con 0x7fd5ec0faad0 2021-11-19T08:35:49.470 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.476+0000 7fd5ea7fc700 1 -- 172.21.15.23:0/3424842393 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5dc0103a0 con 0x7fd5ec0faad0 2021-11-19T08:35:49.471 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.477+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/3424842393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5ec0faad0 msgr2=0x7fd5ec0faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:49.471 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.477+0000 7fd5f1a7b700 1 --2- 172.21.15.23:0/3424842393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5ec0faad0 0x7fd5ec0faef0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fd5dc009960 tx=0x7fd5dc0092d0).stop 2021-11-19T08:35:49.471 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.477+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/3424842393 shutdown_connections 2021-11-19T08:35:49.471 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.477+0000 7fd5f1a7b700 1 --2- 172.21.15.23:0/3424842393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5ec0faad0 0x7fd5ec0faef0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:49.471 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.477+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/3424842393 >> 172.21.15.23:0/3424842393 conn(0x7fd5ec0f6310 msgr2=0x7fd5ec0f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:49.472 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.477+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/3424842393 shutdown_connections 2021-11-19T08:35:49.472 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.477+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/3424842393 wait complete. 2021-11-19T08:35:49.472 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.477+0000 7fd5f1a7b700 1 Processor -- start 2021-11-19T08:35:49.472 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.478+0000 7fd5f1a7b700 1 -- start start 2021-11-19T08:35:49.473 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.478+0000 7fd5f1a7b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5ec0faad0 0x7fd5ec108740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:49.473 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.478+0000 7fd5f1a7b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec108c30 con 0x7fd5ec0faad0 2021-11-19T08:35:49.473 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.478+0000 7fd5eb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5ec0faad0 0x7fd5ec108740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:49.473 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.478+0000 7fd5eb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5ec0faad0 0x7fd5ec108740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46062/0 (socket says 172.21.15.23:46062) 2021-11-19T08:35:49.474 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.478+0000 7fd5eb7fe700 1 -- 172.21.15.23:0/642493181 learned_addr learned my addr 172.21.15.23:0/642493181 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:49.474 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.478+0000 7fd5eb7fe700 1 -- 172.21.15.23:0/642493181 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5dc009570 con 0x7fd5ec0faad0 2021-11-19T08:35:49.474 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.478+0000 7fd5eb7fe700 1 --2- 172.21.15.23:0/642493181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5ec0faad0 0x7fd5ec108740 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fd5dc000f80 tx=0x7fd5dc0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:49.474 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.478+0000 7fd5e3fff700 1 -- 172.21.15.23:0/642493181 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5dc010820 con 0x7fd5ec0faad0 2021-11-19T08:35:49.475 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.478+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/642493181 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ec108e30 con 0x7fd5ec0faad0 2021-11-19T08:35:49.475 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.479+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/642493181 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ec109240 con 0x7fd5ec0faad0 2021-11-19T08:35:49.475 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.479+0000 7fd5e3fff700 1 -- 172.21.15.23:0/642493181 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd5dc010df0 con 0x7fd5ec0faad0 2021-11-19T08:35:49.475 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.479+0000 7fd5e3fff700 1 -- 172.21.15.23:0/642493181 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5dc01ac80 con 0x7fd5ec0faad0 2021-11-19T08:35:49.476 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.479+0000 7fd5e3fff700 1 -- 172.21.15.23:0/642493181 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd5dc00f070 con 0x7fd5ec0faad0 2021-11-19T08:35:49.476 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.479+0000 7fd5e3fff700 1 --2- 172.21.15.23:0/642493181 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fd5cc031fa0 0x7fd5cc034460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:49.476 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.480+0000 7fd5e3fff700 1 -- 172.21.15.23:0/642493181 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd5dc042ca0 con 0x7fd5ec0faad0 2021-11-19T08:35:49.476 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.480+0000 7fd5eaffd700 1 --2- 172.21.15.23:0/642493181 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fd5cc031fa0 0x7fd5cc034460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:49.477 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.480+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/642493181 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d0004fa0 con 0x7fd5ec0faad0 2021-11-19T08:35:49.477 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.480+0000 7fd5eaffd700 1 --2- 172.21.15.23:0/642493181 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fd5cc031fa0 0x7fd5cc034460 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd5d40096a0 tx=0x7fd5d4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:49.478 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.484+0000 7fd5e3fff700 1 -- 172.21.15.23:0/642493181 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5dc01e070 con 0x7fd5ec0faad0 2021-11-19T08:35:49.801 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:49 smithi023 conmon[33490]: audit 2021-11-19T08:35:48.624616+0000 mgr.smithi023.ehlpcl (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:35:49.802 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:49.815+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/642493181 --> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5d00029f0 con 0x7fd5cc031fa0 2021-11-19T08:35:50.525 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Added host 'smithi023' 2021-11-19T08:35:50.525 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:50.537+0000 7fd5e3fff700 1 -- 172.21.15.23:0/642493181 <== mgr.14116 v2:172.21.15.23:6800/1352671878 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fd5d00029f0 con 0x7fd5cc031fa0 2021-11-19T08:35:50.527 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:50.540+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/642493181 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fd5cc031fa0 msgr2=0x7fd5cc034460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:50.527 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:50.540+0000 7fd5f1a7b700 1 --2- 172.21.15.23:0/642493181 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fd5cc031fa0 0x7fd5cc034460 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd5d40096a0 tx=0x7fd5d4006b40).stop 2021-11-19T08:35:50.527 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:50.540+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/642493181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5ec0faad0 msgr2=0x7fd5ec108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:50.528 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:50.540+0000 7fd5f1a7b700 1 --2- 172.21.15.23:0/642493181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5ec0faad0 0x7fd5ec108740 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fd5dc000f80 tx=0x7fd5dc0092d0).stop 2021-11-19T08:35:50.528 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:50.540+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/642493181 shutdown_connections 2021-11-19T08:35:50.528 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:50.540+0000 7fd5f1a7b700 1 --2- 172.21.15.23:0/642493181 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fd5cc031fa0 0x7fd5cc034460 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:50.528 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:50.540+0000 7fd5f1a7b700 1 --2- 172.21.15.23:0/642493181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5ec0faad0 0x7fd5ec108740 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:50.528 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:50.540+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/642493181 >> 172.21.15.23:0/642493181 conn(0x7fd5ec0f6310 msgr2=0x7fd5ec0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:50.529 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:50.541+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/642493181 shutdown_connections 2021-11-19T08:35:50.529 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:50.541+0000 7fd5f1a7b700 1 -- 172.21.15.23:0/642493181 wait complete. 2021-11-19T08:35:50.936 INFO:teuthology.orchestra.run.smithi023.stderr:Deploying mon service with default placement... 2021-11-19T08:35:50.937 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2021-11-19T08:35:51.404 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.416+0000 7fab85f89700 1 Processor -- start 2021-11-19T08:35:51.405 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.417+0000 7fab85f89700 1 -- start start 2021-11-19T08:35:51.405 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.417+0000 7fab85f89700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab800fb9a0 0x7fab800fbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:51.405 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.417+0000 7fab85f89700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab800fc2b0 con 0x7fab800fb9a0 2021-11-19T08:35:51.405 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.417+0000 7fab84f87700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab800fb9a0 0x7fab800fbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:51.406 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.418+0000 7fab84f87700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab800fb9a0 0x7fab800fbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46074/0 (socket says 172.21.15.23:46074) 2021-11-19T08:35:51.406 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.418+0000 7fab84f87700 1 -- 172.21.15.23:0/479249804 learned_addr learned my addr 172.21.15.23:0/479249804 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:51.406 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.418+0000 7fab84f87700 1 -- 172.21.15.23:0/479249804 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab800fc3f0 con 0x7fab800fb9a0 2021-11-19T08:35:51.407 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.418+0000 7fab84f87700 1 --2- 172.21.15.23:0/479249804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab800fb9a0 0x7fab800fbdc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fab7400b5f0 tx=0x7fab74006ee0).ready entity=mon.0 client_cookie=ab601605b50deb11 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:51.407 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.418+0000 7fab7f7fe700 1 -- 172.21.15.23:0/479249804 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab7400d600 con 0x7fab800fb9a0 2021-11-19T08:35:51.407 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.418+0000 7fab7f7fe700 1 -- 172.21.15.23:0/479249804 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fab7400dbd0 con 0x7fab800fb9a0 2021-11-19T08:35:51.407 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.418+0000 7fab7f7fe700 1 -- 172.21.15.23:0/479249804 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab74011c00 con 0x7fab800fb9a0 2021-11-19T08:35:51.411 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.419+0000 7fab85f89700 1 -- 172.21.15.23:0/479249804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab800fb9a0 msgr2=0x7fab800fbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:51.412 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.419+0000 7fab85f89700 1 --2- 172.21.15.23:0/479249804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab800fb9a0 0x7fab800fbdc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fab7400b5f0 tx=0x7fab74006ee0).stop 2021-11-19T08:35:51.412 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.419+0000 7fab85f89700 1 -- 172.21.15.23:0/479249804 shutdown_connections 2021-11-19T08:35:51.412 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.419+0000 7fab85f89700 1 --2- 172.21.15.23:0/479249804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab800fb9a0 0x7fab800fbdc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:51.413 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.419+0000 7fab85f89700 1 -- 172.21.15.23:0/479249804 >> 172.21.15.23:0/479249804 conn(0x7fab800f71e0 msgr2=0x7fab800f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:51.413 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.419+0000 7fab85f89700 1 -- 172.21.15.23:0/479249804 shutdown_connections 2021-11-19T08:35:51.413 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.419+0000 7fab85f89700 1 -- 172.21.15.23:0/479249804 wait complete. 2021-11-19T08:35:51.413 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.419+0000 7fab85f89700 1 Processor -- start 2021-11-19T08:35:51.414 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.420+0000 7fab85f89700 1 -- start start 2021-11-19T08:35:51.414 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.420+0000 7fab85f89700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab800fb9a0 0x7fab80071d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:51.414 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.420+0000 7fab85f89700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab800701f0 con 0x7fab800fb9a0 2021-11-19T08:35:51.414 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.420+0000 7fab84f87700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab800fb9a0 0x7fab80071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:51.415 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.420+0000 7fab84f87700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab800fb9a0 0x7fab80071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46076/0 (socket says 172.21.15.23:46076) 2021-11-19T08:35:51.415 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.420+0000 7fab84f87700 1 -- 172.21.15.23:0/4262437256 learned_addr learned my addr 172.21.15.23:0/4262437256 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:51.415 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.420+0000 7fab84f87700 1 -- 172.21.15.23:0/4262437256 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7400b200 con 0x7fab800fb9a0 2021-11-19T08:35:51.415 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.420+0000 7fab84f87700 1 --2- 172.21.15.23:0/4262437256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab800fb9a0 0x7fab80071d70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fab7400b1d0 tx=0x7fab74006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:51.416 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.421+0000 7fab7d7fa700 1 -- 172.21.15.23:0/4262437256 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab7401b410 con 0x7fab800fb9a0 2021-11-19T08:35:51.416 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.421+0000 7fab7d7fa700 1 -- 172.21.15.23:0/4262437256 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fab7401b9e0 con 0x7fab800fb9a0 2021-11-19T08:35:51.416 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.421+0000 7fab85f89700 1 -- 172.21.15.23:0/4262437256 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab80070390 con 0x7fab800fb9a0 2021-11-19T08:35:51.416 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.421+0000 7fab7d7fa700 1 -- 172.21.15.23:0/4262437256 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab7401a840 con 0x7fab800fb9a0 2021-11-19T08:35:51.417 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.421+0000 7fab85f89700 1 -- 172.21.15.23:0/4262437256 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab800707a0 con 0x7fab800fb9a0 2021-11-19T08:35:51.417 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.422+0000 7fab85f89700 1 -- 172.21.15.23:0/4262437256 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab80045ad0 con 0x7fab800fb9a0 2021-11-19T08:35:51.417 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.422+0000 7fab7d7fa700 1 -- 172.21.15.23:0/4262437256 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fab7400f070 con 0x7fab800fb9a0 2021-11-19T08:35:51.417 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.422+0000 7fab7d7fa700 1 --2- 172.21.15.23:0/4262437256 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fab6402fdb0 0x7fab64032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.422+0000 7fab7d7fa700 1 -- 172.21.15.23:0/4262437256 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fab740411a0 con 0x7fab800fb9a0 2021-11-19T08:35:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.423+0000 7fab7ffff700 1 --2- 172.21.15.23:0/4262437256 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fab6402fdb0 0x7fab64032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.423+0000 7fab7ffff700 1 --2- 172.21.15.23:0/4262437256 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fab6402fdb0 0x7fab64032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fab6c0096a0 tx=0x7fab6c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:51.419 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.427+0000 7fab7d7fa700 1 -- 172.21.15.23:0/4262437256 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab74019950 con 0x7fab800fb9a0 2021-11-19T08:35:51.753 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:51 smithi023 conmon[33490]: audit 2021-11-19T08:35:49.817098+0000 mgr.smithi023.ehlpcl (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:35:51.753 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:51 smithi023 conmon[33490]: audit 2021-11-19T08:35:50.537813+0000 mon.smithi023 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:51.754 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:51 smithi023 conmon[33490]: cephadm 2021-11-19T08:35:50.538173+0000 mgr.smithi023.ehlpcl (mgr.14116) 9 : cephadm [INF] Added host smithi023 2021-11-19T08:35:51.754 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:51 smithi023 conmon[33490]: audit 2021-11-19T08:35:50.538466+0000 mon.smithi023 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:35:51.754 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.767+0000 7fab85f89700 1 -- 172.21.15.23:0/4262437256 --> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fab800fd890 con 0x7fab6402fdb0 2021-11-19T08:35:51.758 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.772+0000 7fab7d7fa700 1 -- 172.21.15.23:0/4262437256 <== mgr.14116 v2:172.21.15.23:6800/1352671878 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fab800fd890 con 0x7fab6402fdb0 2021-11-19T08:35:51.758 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Scheduled mon update... 2021-11-19T08:35:51.761 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.774+0000 7fab85f89700 1 -- 172.21.15.23:0/4262437256 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fab6402fdb0 msgr2=0x7fab64032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:51.762 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.774+0000 7fab85f89700 1 --2- 172.21.15.23:0/4262437256 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fab6402fdb0 0x7fab64032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fab6c0096a0 tx=0x7fab6c006b40).stop 2021-11-19T08:35:51.762 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.774+0000 7fab85f89700 1 -- 172.21.15.23:0/4262437256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab800fb9a0 msgr2=0x7fab80071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:51.762 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.774+0000 7fab85f89700 1 --2- 172.21.15.23:0/4262437256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab800fb9a0 0x7fab80071d70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fab7400b1d0 tx=0x7fab74006ee0).stop 2021-11-19T08:35:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.774+0000 7fab85f89700 1 -- 172.21.15.23:0/4262437256 shutdown_connections 2021-11-19T08:35:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.774+0000 7fab85f89700 1 --2- 172.21.15.23:0/4262437256 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7fab6402fdb0 0x7fab64032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.774+0000 7fab85f89700 1 --2- 172.21.15.23:0/4262437256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab800fb9a0 0x7fab80071d70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.774+0000 7fab85f89700 1 -- 172.21.15.23:0/4262437256 >> 172.21.15.23:0/4262437256 conn(0x7fab800f71e0 msgr2=0x7fab800f8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.774+0000 7fab85f89700 1 -- 172.21.15.23:0/4262437256 shutdown_connections 2021-11-19T08:35:51.764 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:51.775+0000 7fab85f89700 1 -- 172.21.15.23:0/4262437256 wait complete. 2021-11-19T08:35:52.074 INFO:teuthology.orchestra.run.smithi023.stderr:Deploying mgr service with default placement... 2021-11-19T08:35:52.075 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2021-11-19T08:35:52.707 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.718+0000 7f887f054700 1 Processor -- start 2021-11-19T08:35:52.707 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.718+0000 7f887f054700 1 -- start start 2021-11-19T08:35:52.707 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.719+0000 7f887f054700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88780fd0c0 0x7f88780fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:52.707 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.719+0000 7f887f054700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88780fd9d0 con 0x7f88780fd0c0 2021-11-19T08:35:52.708 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.719+0000 7f887e052700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88780fd0c0 0x7f88780fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:52.708 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.719+0000 7f887e052700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88780fd0c0 0x7f88780fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46082/0 (socket says 172.21.15.23:46082) 2021-11-19T08:35:52.708 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.719+0000 7f887e052700 1 -- 172.21.15.23:0/70858163 learned_addr learned my addr 172.21.15.23:0/70858163 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:52.708 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.719+0000 7f887e052700 1 -- 172.21.15.23:0/70858163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88780fdb10 con 0x7f88780fd0c0 2021-11-19T08:35:52.709 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.719+0000 7f887e052700 1 --2- 172.21.15.23:0/70858163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88780fd0c0 0x7f88780fd4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f886800b5f0 tx=0x7f8868006ee0).ready entity=mon.0 client_cookie=fa5677271bfb7ad7 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:52.709 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.720+0000 7f887d050700 1 -- 172.21.15.23:0/70858163 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f886800d600 con 0x7f88780fd0c0 2021-11-19T08:35:52.709 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.720+0000 7f887d050700 1 -- 172.21.15.23:0/70858163 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f886800dbd0 con 0x7f88780fd0c0 2021-11-19T08:35:52.709 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.720+0000 7f887d050700 1 -- 172.21.15.23:0/70858163 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8868011c00 con 0x7f88780fd0c0 2021-11-19T08:35:52.710 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.720+0000 7f887f054700 1 -- 172.21.15.23:0/70858163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88780fd0c0 msgr2=0x7f88780fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:52.710 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.720+0000 7f887f054700 1 --2- 172.21.15.23:0/70858163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88780fd0c0 0x7f88780fd4e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f886800b5f0 tx=0x7f8868006ee0).stop 2021-11-19T08:35:52.710 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.720+0000 7f887f054700 1 -- 172.21.15.23:0/70858163 shutdown_connections 2021-11-19T08:35:52.710 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.720+0000 7f887f054700 1 --2- 172.21.15.23:0/70858163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88780fd0c0 0x7f88780fd4e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:52.710 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.720+0000 7f887f054700 1 -- 172.21.15.23:0/70858163 >> 172.21.15.23:0/70858163 conn(0x7f88780f5880 msgr2=0x7f88780f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:52.711 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.721+0000 7f887f054700 1 -- 172.21.15.23:0/70858163 shutdown_connections 2021-11-19T08:35:52.711 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.721+0000 7f887f054700 1 -- 172.21.15.23:0/70858163 wait complete. 2021-11-19T08:35:52.712 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.721+0000 7f887f054700 1 Processor -- start 2021-11-19T08:35:52.713 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.722+0000 7f887f054700 1 -- start start 2021-11-19T08:35:52.713 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.722+0000 7f887f054700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88780fd0c0 0x7f887810a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:52.713 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.722+0000 7f887f054700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887810acd0 con 0x7f88780fd0c0 2021-11-19T08:35:52.713 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.722+0000 7f887e052700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88780fd0c0 0x7f887810a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:52.713 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.722+0000 7f887e052700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88780fd0c0 0x7f887810a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46084/0 (socket says 172.21.15.23:46084) 2021-11-19T08:35:52.714 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.722+0000 7f887e052700 1 -- 172.21.15.23:0/2755898590 learned_addr learned my addr 172.21.15.23:0/2755898590 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:52.714 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.722+0000 7f887e052700 1 -- 172.21.15.23:0/2755898590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886800b200 con 0x7f88780fd0c0 2021-11-19T08:35:52.714 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.722+0000 7f887e052700 1 --2- 172.21.15.23:0/2755898590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88780fd0c0 0x7f887810a7e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f8868006c70 tx=0x7f8868006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:52.714 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.722+0000 7f8866ffd700 1 -- 172.21.15.23:0/2755898590 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f886800d9f0 con 0x7f88780fd0c0 2021-11-19T08:35:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.723+0000 7f887f054700 1 -- 172.21.15.23:0/2755898590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f887810aed0 con 0x7f88780fd0c0 2021-11-19T08:35:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.723+0000 7f8866ffd700 1 -- 172.21.15.23:0/2755898590 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f886801a9f0 con 0x7f88780fd0c0 2021-11-19T08:35:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.723+0000 7f8866ffd700 1 -- 172.21.15.23:0/2755898590 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f886801b890 con 0x7f88780fd0c0 2021-11-19T08:35:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.723+0000 7f887f054700 1 -- 172.21.15.23:0/2755898590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887810b2e0 con 0x7f88780fd0c0 2021-11-19T08:35:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.723+0000 7f8866ffd700 1 -- 172.21.15.23:0/2755898590 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f886800f070 con 0x7f88780fd0c0 2021-11-19T08:35:52.716 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.724+0000 7f8866ffd700 1 --2- 172.21.15.23:0/2755898590 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f886002fd50 0x7f8860032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:52.716 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.724+0000 7f8866ffd700 1 -- 172.21.15.23:0/2755898590 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8868041410 con 0x7f88780fd0c0 2021-11-19T08:35:52.716 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.724+0000 7f887f054700 1 -- 172.21.15.23:0/2755898590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8858004fa0 con 0x7f88780fd0c0 2021-11-19T08:35:52.716 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.724+0000 7f887d851700 1 --2- 172.21.15.23:0/2755898590 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f886002fd50 0x7f8860032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:52.717 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.724+0000 7f887d851700 1 --2- 172.21.15.23:0/2755898590 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f886002fd50 0x7f8860032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f88740096a0 tx=0x7f8874006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:52.717 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.728+0000 7f8866ffd700 1 -- 172.21.15.23:0/2755898590 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8868016070 con 0x7f88780fd0c0 2021-11-19T08:35:52.980 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:52 smithi023 conmon[33490]: audit 2021-11-19T08:35:51. 2021-11-19T08:35:52.981 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:52 smithi023 conmon[33490]: 769254+0000 mgr.smithi023.ehlpcl (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:35:52.981 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:52 smithi023 conmon[33490]: cephadm 2021-11-19T08:35:51.770267+0000 mgr.smithi023.ehlpcl (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2021-11-19T08:35:52.981 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:52 smithi023 conmon[33490]: audit 2021-11-19T08:35:51.772803+0000 mon.smithi023 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:52.981 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:52.994+0000 7f8866ffd700 1 -- 172.21.15.23:0/2755898590 <== mon.0 v2:172.21.15.23:3300/0 7 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8868025460 con 0x7f88780fd0c0 2021-11-19T08:35:53.067 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.081+0000 7f887f054700 1 -- 172.21.15.23:0/2755898590 --> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f8858002a50 con 0x7f886002fd50 2021-11-19T08:35:53.071 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.085+0000 7f8866ffd700 1 -- 172.21.15.23:0/2755898590 <== mgr.14116 v2:172.21.15.23:6800/1352671878 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8858002a50 con 0x7f886002fd50 2021-11-19T08:35:53.071 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Scheduled mgr update... 2021-11-19T08:35:53.073 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.087+0000 7f887f054700 1 -- 172.21.15.23:0/2755898590 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f886002fd50 msgr2=0x7f8860032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:53.073 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.087+0000 7f887f054700 1 --2- 172.21.15.23:0/2755898590 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f886002fd50 0x7f8860032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f88740096a0 tx=0x7f8874006b40).stop 2021-11-19T08:35:53.073 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.087+0000 7f887f054700 1 -- 172.21.15.23:0/2755898590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88780fd0c0 msgr2=0x7f887810a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:53.074 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.087+0000 7f887f054700 1 --2- 172.21.15.23:0/2755898590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88780fd0c0 0x7f887810a7e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f8868006c70 tx=0x7f8868006ee0).stop 2021-11-19T08:35:53.074 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.087+0000 7f887f054700 1 -- 172.21.15.23:0/2755898590 shutdown_connections 2021-11-19T08:35:53.074 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.087+0000 7f887f054700 1 --2- 172.21.15.23:0/2755898590 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f886002fd50 0x7f8860032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:53.075 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.087+0000 7f887f054700 1 --2- 172.21.15.23:0/2755898590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88780fd0c0 0x7f887810a7e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:53.075 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.087+0000 7f887f054700 1 -- 172.21.15.23:0/2755898590 >> 172.21.15.23:0/2755898590 conn(0x7f88780f5880 msgr2=0x7f88780ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:53.075 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.087+0000 7f887f054700 1 -- 172.21.15.23:0/2755898590 shutdown_connections 2021-11-19T08:35:53.076 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.087+0000 7f887f054700 1 -- 172.21.15.23:0/2755898590 wait complete. 2021-11-19T08:35:53.411 INFO:teuthology.orchestra.run.smithi023.stderr:Deploying crash service with default placement... 2021-11-19T08:35:53.411 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2021-11-19T08:35:53.906 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.917+0000 7f9196f33700 1 Processor -- start 2021-11-19T08:35:53.906 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.917+0000 7f9196f33700 1 -- start start 2021-11-19T08:35:53.906 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.917+0000 7f9196f33700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900fc210 0x7f91900fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.917+0000 7f9196f33700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91900fcb20 con 0x7f91900fc210 2021-11-19T08:35:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.917+0000 7f9194ccf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900fc210 0x7f91900fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.917+0000 7f9194ccf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900fc210 0x7f91900fc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46088/0 (socket says 172.21.15.23:46088) 2021-11-19T08:35:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.917+0000 7f9194ccf700 1 -- 172.21.15.23:0/2158559794 learned_addr learned my addr 172.21.15.23:0/2158559794 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.917+0000 7f9194ccf700 1 -- 172.21.15.23:0/2158559794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91900fcc60 con 0x7f91900fc210 2021-11-19T08:35:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.917+0000 7f9194ccf700 1 --2- 172.21.15.23:0/2158559794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900fc210 0x7f91900fc630 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f9188009960 tx=0x7f91880092d0).ready entity=mon.0 client_cookie=6e1d123134b80e5b server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.918+0000 7f918f7fe700 1 -- 172.21.15.23:0/2158559794 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f918800b9c0 con 0x7f91900fc210 2021-11-19T08:35:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.918+0000 7f918f7fe700 1 -- 172.21.15.23:0/2158559794 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9188011400 con 0x7f91900fc210 2021-11-19T08:35:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.918+0000 7f918f7fe700 1 -- 172.21.15.23:0/2158559794 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9188010580 con 0x7f91900fc210 2021-11-19T08:35:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.918+0000 7f9196f33700 1 -- 172.21.15.23:0/2158559794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900fc210 msgr2=0x7f91900fc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.918+0000 7f9196f33700 1 --2- 172.21.15.23:0/2158559794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900fc210 0x7f91900fc630 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f9188009960 tx=0x7f91880092d0).stop 2021-11-19T08:35:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.918+0000 7f9196f33700 1 -- 172.21.15.23:0/2158559794 shutdown_connections 2021-11-19T08:35:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.918+0000 7f9196f33700 1 --2- 172.21.15.23:0/2158559794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900fc210 0x7f91900fc630 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.918+0000 7f9196f33700 1 -- 172.21.15.23:0/2158559794 >> 172.21.15.23:0/2158559794 conn(0x7f91900f5a80 msgr2=0x7f91900f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.918+0000 7f9196f33700 1 -- 172.21.15.23:0/2158559794 shutdown_connections 2021-11-19T08:35:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.918+0000 7f9196f33700 1 -- 172.21.15.23:0/2158559794 wait complete. 2021-11-19T08:35:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.919+0000 7f9196f33700 1 Processor -- start 2021-11-19T08:35:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.919+0000 7f9196f33700 1 -- start start 2021-11-19T08:35:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.919+0000 7f9196f33700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900fc210 0x7f919010cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.919+0000 7f9196f33700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919010d0c0 con 0x7f91900fc210 2021-11-19T08:35:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.920+0000 7f9194ccf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900fc210 0x7f919010cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:53.912 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.920+0000 7f9194ccf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900fc210 0x7f919010cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46090/0 (socket says 172.21.15.23:46090) 2021-11-19T08:35:53.912 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.920+0000 7f9194ccf700 1 -- 172.21.15.23:0/1093048626 learned_addr learned my addr 172.21.15.23:0/1093048626 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:53.912 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.920+0000 7f9194ccf700 1 -- 172.21.15.23:0/1093048626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9188009570 con 0x7f91900fc210 2021-11-19T08:35:53.912 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.920+0000 7f9194ccf700 1 --2- 172.21.15.23:0/1093048626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900fc210 0x7f919010cbd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f9188006dc0 tx=0x7f91880092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:53.914 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.920+0000 7f918d7fa700 1 -- 172.21.15.23:0/1093048626 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91880103a0 con 0x7f91900fc210 2021-11-19T08:35:53.915 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.920+0000 7f918d7fa700 1 -- 172.21.15.23:0/1093048626 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9188010cf0 con 0x7f91900fc210 2021-11-19T08:35:53.915 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.920+0000 7f9196f33700 1 -- 172.21.15.23:0/1093048626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919010d2c0 con 0x7f91900fc210 2021-11-19T08:35:53.915 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.920+0000 7f9196f33700 1 -- 172.21.15.23:0/1093048626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919010d6d0 con 0x7f91900fc210 2021-11-19T08:35:53.915 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.920+0000 7f918d7fa700 1 -- 172.21.15.23:0/1093048626 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f918801baf0 con 0x7f91900fc210 2021-11-19T08:35:53.916 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.921+0000 7f918d7fa700 1 -- 172.21.15.23:0/1093048626 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f918800f070 con 0x7f91900fc210 2021-11-19T08:35:53.916 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.921+0000 7f918d7fa700 1 --2- 172.21.15.23:0/1093048626 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f917002fe00 0x7f91700322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:53.916 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.921+0000 7f918d7fa700 1 -- 172.21.15.23:0/1093048626 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9188042720 con 0x7f91900fc210 2021-11-19T08:35:53.916 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.921+0000 7f918ffff700 1 --2- 172.21.15.23:0/1093048626 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f917002fe00 0x7f91700322c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:53.916 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.922+0000 7f9196f33700 1 -- 172.21.15.23:0/1093048626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9174004fa0 con 0x7f91900fc210 2021-11-19T08:35:53.917 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.922+0000 7f918ffff700 1 --2- 172.21.15.23:0/1093048626 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f917002fe00 0x7f91700322c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f91800096a0 tx=0x7f9180006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:53.917 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:53.926+0000 7f918d7fa700 1 -- 172.21.15.23:0/1093048626 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f918801be30 con 0x7f91900fc210 2021-11-19T08:35:54.258 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:54.272+0000 7f9196f33700 1 -- 172.21.15.23:0/1093048626 --> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f9174000bc0 con 0x7f917002fe00 2021-11-19T08:35:54.260 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:53 smithi023 conmon[33490]: audit 2021-11-19T08:35: 2021-11-19T08:35:54.260 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:53 smithi023 conmon[33490]: 52.995406+0000 mon.smithi023 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:54.260 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:53 smithi023 conmon[33490]: audit 2021-11-19T08:35:53.082775+0000 mgr.smithi023.ehlpcl (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:35:54.261 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:53 smithi023 conmon[33490]: cephadm 2021-11-19T 2021-11-19T08:35:54.261 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:53 smithi023 conmon[33490]: 08:35:53.083338+0000 mgr.smithi023.ehlpcl (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2021-11-19T08:35:54.261 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:53 smithi023 conmon[33490]: audit 2021-11- 2021-11-19T08:35:54.262 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:53 smithi023 conmon[33490]: 19T08:35:53.085596+0000 mon.smithi023 ( 2021-11-19T08:35:54.262 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:53 smithi023 conmon[33490]: mon.0) 59 : audit 2021-11-19T08:35:54.262 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:53 smithi023 conmon[33490]: [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:54.262 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:53 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:35:54.263 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:53 smithi023 conmon[33490]: -19T08:35:53.510954+0000 mon.smithi023 2021-11-19T08:35:54.263 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:53 smithi023 conmon[33490]: (mon.0) 60 : audit 2021-11-19T08:35:54.263 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:53 smithi023 conmon[33490]: [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:54.264 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:54.277+0000 7f918d7fa700 1 -- 172.21.15.23:0/1093048626 <== mgr.14116 v2:172.21.15.23:6800/1352671878 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f9174000bc0 con 0x7f917002fe00 2021-11-19T08:35:54.264 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Scheduled crash update... 2021-11-19T08:35:54.265 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:54.278+0000 7f9196f33700 1 -- 172.21.15.23:0/1093048626 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f917002fe00 msgr2=0x7f91700322c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:54.266 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:54.278+0000 7f9196f33700 1 --2- 172.21.15.23:0/1093048626 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f917002fe00 0x7f91700322c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f91800096a0 tx=0x7f9180006b40).stop 2021-11-19T08:35:54.266 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:54.278+0000 7f9196f33700 1 -- 172.21.15.23:0/1093048626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900fc210 msgr2=0x7f919010cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:54.266 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:54.278+0000 7f9196f33700 1 --2- 172.21.15.23:0/1093048626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900fc210 0x7f919010cbd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f9188006dc0 tx=0x7f91880092d0).stop 2021-11-19T08:35:54.266 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:54.279+0000 7f9196f33700 1 -- 172.21.15.23:0/1093048626 shutdown_connections 2021-11-19T08:35:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:54.279+0000 7f9196f33700 1 --2- 172.21.15.23:0/1093048626 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f917002fe00 0x7f91700322c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:54.279+0000 7f9196f33700 1 --2- 172.21.15.23:0/1093048626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900fc210 0x7f919010cbd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:54.279+0000 7f9196f33700 1 -- 172.21.15.23:0/1093048626 >> 172.21.15.23:0/1093048626 conn(0x7f91900f5a80 msgr2=0x7f91900f75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:54.279+0000 7f9196f33700 1 -- 172.21.15.23:0/1093048626 shutdown_connections 2021-11-19T08:35:54.268 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:54.279+0000 7f9196f33700 1 -- 172.21.15.23:0/1093048626 wait complete. 2021-11-19T08:35:54.747 INFO:teuthology.orchestra.run.smithi023.stderr:Enabling mgr prometheus module... 2021-11-19T08:35:54.748 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2021-11-19T08:35:55.305 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.318+0000 7f4191c53700 1 Processor -- start 2021-11-19T08:35:55.306 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.318+0000 7f4191c53700 1 -- start start 2021-11-19T08:35:55.306 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.319+0000 7f4191c53700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c06fc90 0x7f418c0fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:55.306 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.319+0000 7f4191c53700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0fed60 con 0x7f418c06fc90 2021-11-19T08:35:55.306 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.319+0000 7f418b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c06fc90 0x7f418c0fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:55.307 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.319+0000 7f418b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c06fc90 0x7f418c0fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46094/0 (socket says 172.21.15.23:46094) 2021-11-19T08:35:55.307 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.319+0000 7f418b7fe700 1 -- 172.21.15.23:0/2054321379 learned_addr learned my addr 172.21.15.23:0/2054321379 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:55.307 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.319+0000 7f418b7fe700 1 -- 172.21.15.23:0/2054321379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c0feea0 con 0x7f418c06fc90 2021-11-19T08:35:55.307 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.320+0000 7f418b7fe700 1 --2- 172.21.15.23:0/2054321379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c06fc90 0x7f418c0fe7e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f417c00b5f0 tx=0x7f417c006ee0).ready entity=mon.0 client_cookie=a2530ddb6c1bc5ff server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:55.311 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.320+0000 7f418a7fc700 1 -- 172.21.15.23:0/2054321379 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f417c00d600 con 0x7f418c06fc90 2021-11-19T08:35:55.312 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.320+0000 7f418a7fc700 1 -- 172.21.15.23:0/2054321379 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f417c00dc10 con 0x7f418c06fc90 2021-11-19T08:35:55.312 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.320+0000 7f418a7fc700 1 -- 172.21.15.23:0/2054321379 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f417c011cd0 con 0x7f418c06fc90 2021-11-19T08:35:55.312 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.321+0000 7f4191c53700 1 -- 172.21.15.23:0/2054321379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c06fc90 msgr2=0x7f418c0fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:55.313 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.321+0000 7f4191c53700 1 --2- 172.21.15.23:0/2054321379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c06fc90 0x7f418c0fe7e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f417c00b5f0 tx=0x7f417c006ee0).stop 2021-11-19T08:35:55.313 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.321+0000 7f4191c53700 1 -- 172.21.15.23:0/2054321379 shutdown_connections 2021-11-19T08:35:55.313 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.321+0000 7f4191c53700 1 --2- 172.21.15.23:0/2054321379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c06fc90 0x7f418c0fe7e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:55.313 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.321+0000 7f4191c53700 1 -- 172.21.15.23:0/2054321379 >> 172.21.15.23:0/2054321379 conn(0x7f418c0f6280 msgr2=0x7f418c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:55.314 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.321+0000 7f4191c53700 1 -- 172.21.15.23:0/2054321379 shutdown_connections 2021-11-19T08:35:55.314 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.321+0000 7f4191c53700 1 -- 172.21.15.23:0/2054321379 wait complete. 2021-11-19T08:35:55.314 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.321+0000 7f4191c53700 1 Processor -- start 2021-11-19T08:35:55.314 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.322+0000 7f4191c53700 1 -- start start 2021-11-19T08:35:55.314 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.322+0000 7f4191c53700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c06fc90 0x7f418c0fc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:55.315 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.322+0000 7f4191c53700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0fa6e0 con 0x7f418c06fc90 2021-11-19T08:35:55.315 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.322+0000 7f418b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c06fc90 0x7f418c0fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:55.315 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.322+0000 7f418b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c06fc90 0x7f418c0fc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46096/0 (socket says 172.21.15.23:46096) 2021-11-19T08:35:55.315 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.322+0000 7f418b7fe700 1 -- 172.21.15.23:0/2013927429 learned_addr learned my addr 172.21.15.23:0/2013927429 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:55.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.322+0000 7f418b7fe700 1 -- 172.21.15.23:0/2013927429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c00b200 con 0x7f418c06fc90 2021-11-19T08:35:55.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.322+0000 7f418b7fe700 1 --2- 172.21.15.23:0/2013927429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c06fc90 0x7f418c0fc260 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f417c00dfb0 tx=0x7f417c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:55.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.322+0000 7f4173fff700 1 -- 172.21.15.23:0/2013927429 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f417c011df0 con 0x7f418c06fc90 2021-11-19T08:35:55.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.323+0000 7f4191c53700 1 -- 172.21.15.23:0/2013927429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418c0fa880 con 0x7f418c06fc90 2021-11-19T08:35:55.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.323+0000 7f4173fff700 1 -- 172.21.15.23:0/2013927429 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f417c01b8c0 con 0x7f418c06fc90 2021-11-19T08:35:55.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.323+0000 7f4173fff700 1 -- 172.21.15.23:0/2013927429 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f417c01a7c0 con 0x7f418c06fc90 2021-11-19T08:35:55.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.323+0000 7f4191c53700 1 -- 172.21.15.23:0/2013927429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418c0fac90 con 0x7f418c06fc90 2021-11-19T08:35:55.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.323+0000 7f4173fff700 1 -- 172.21.15.23:0/2013927429 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f417c00f070 con 0x7f418c06fc90 2021-11-19T08:35:55.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.324+0000 7f4173fff700 1 --2- 172.21.15.23:0/2013927429 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f417802fd50 0x7f4178032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:55.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.324+0000 7f4173fff700 1 -- 172.21.15.23:0/2013927429 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f417c041000 con 0x7f418c06fc90 2021-11-19T08:35:55.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.324+0000 7f4191c53700 1 -- 172.21.15.23:0/2013927429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418c045ad0 con 0x7f418c06fc90 2021-11-19T08:35:55.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.324+0000 7f418affd700 1 --2- 172.21.15.23:0/2013927429 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f417802fd50 0x7f4178032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:55.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.324+0000 7f418affd700 1 --2- 172.21.15.23:0/2013927429 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f417802fd50 0x7f4178032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f41800096a0 tx=0x7f4180006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:55.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.328+0000 7f4173fff700 1 -- 172.21.15.23:0/2013927429 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f417c016070 con 0x7f418c06fc90 2021-11-19T08:35:55.557 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:55 smithi023 conmon[33490]: audit 2021-11-19T08:35:54.274062+0000 mgr.smithi023.ehlpcl (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:35:55.557 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:55 smithi023 conmon[33490]: cephadm 2021-11-19T08:35:54.274995+0000 mgr.smithi023.ehlpcl (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2021-11-19T08:35:55.558 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:55 smithi023 conmon[33490]: audit 2021-11-19T08:35:54.277600+0000 mon.smithi023 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:55.558 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:55 smithi023 conmon[33490]: audit 2021-11-19T08:35:55.074946+0000 mon.smithi023 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.23:0/666628131' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:35:55.678 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:55.691+0000 7f4191c53700 1 -- 172.21.15.23:0/2013927429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f418c104860 con 0x7f418c06fc90 2021-11-19T08:35:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:56.287+0000 7f4173fff700 1 -- 172.21.15.23:0/2013927429 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f417c024790 con 0x7f418c06fc90 2021-11-19T08:35:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:56.288+0000 7f4173fff700 1 -- 172.21.15.23:0/2013927429 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f417c02e660 con 0x7f418c06fc90 2021-11-19T08:35:56.277 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:56.291+0000 7f4191c53700 1 -- 172.21.15.23:0/2013927429 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f417802fd50 msgr2=0x7f4178032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:56.278 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:56.291+0000 7f4191c53700 1 --2- 172.21.15.23:0/2013927429 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f417802fd50 0x7f4178032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f41800096a0 tx=0x7f4180006b40).stop 2021-11-19T08:35:56.278 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:56.291+0000 7f4191c53700 1 -- 172.21.15.23:0/2013927429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c06fc90 msgr2=0x7f418c0fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:56.279 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:56.291+0000 7f4191c53700 1 --2- 172.21.15.23:0/2013927429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c06fc90 0x7f418c0fc260 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f417c00dfb0 tx=0x7f417c006ee0).stop 2021-11-19T08:35:56.279 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:56.291+0000 7f4191c53700 1 -- 172.21.15.23:0/2013927429 shutdown_connections 2021-11-19T08:35:56.280 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:56.291+0000 7f4191c53700 1 --2- 172.21.15.23:0/2013927429 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f417802fd50 0x7f4178032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:56.281 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:56.291+0000 7f4191c53700 1 --2- 172.21.15.23:0/2013927429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c06fc90 0x7f418c0fc260 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0x7f417c00dfb0 tx=0x7f417c006ee0).stop 2021-11-19T08:35:56.281 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:56.291+0000 7f4191c53700 1 -- 172.21.15.23:0/2013927429 >> 172.21.15.23:0/2013927429 conn(0x7f418c0f6280 msgr2=0x7f418c0f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:56.282 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:56.291+0000 7f4191c53700 1 -- 172.21.15.23:0/2013927429 shutdown_connections 2021-11-19T08:35:56.282 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:56.291+0000 7f4191c53700 1 -- 172.21.15.23:0/2013927429 wait complete. 2021-11-19T08:35:56.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:56 smithi023 conmon[33490]: audit 2021-11-19T08:35:55.693101+0000 mon.smithi023 (mon.0) 63 : audit [INF] 2021-11-19T08:35:56.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:56 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/2013927429' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2021-11-19T08:35:56.736 INFO:teuthology.orchestra.run.smithi023.stderr:Deploying prometheus service with default placement... 2021-11-19T08:35:56.737 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2021-11-19T08:35:57.305 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.316+0000 7f6fff7ce700 1 Processor -- start 2021-11-19T08:35:57.306 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.317+0000 7f6fff7ce700 1 -- start start 2021-11-19T08:35:57.306 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.317+0000 7f6fff7ce700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ff80faad0 0x7f6ff80faef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:57.306 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.317+0000 7f6fff7ce700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff80fb3e0 con 0x7f6ff80faad0 2021-11-19T08:35:57.306 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.317+0000 7f6ffd56a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ff80faad0 0x7f6ff80faef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:57.307 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.317+0000 7f6ffd56a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ff80faad0 0x7f6ff80faef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46110/0 (socket says 172.21.15.23:46110) 2021-11-19T08:35:57.307 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.317+0000 7f6ffd56a700 1 -- 172.21.15.23:0/302288322 learned_addr learned my addr 172.21.15.23:0/302288322 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:57.307 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.317+0000 7f6ffd56a700 1 -- 172.21.15.23:0/302288322 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff80fb520 con 0x7f6ff80faad0 2021-11-19T08:35:57.307 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.317+0000 7f6ffd56a700 1 --2- 172.21.15.23:0/302288322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ff80faad0 0x7f6ff80faef0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f6fe8009960 tx=0x7f6fe80092d0).ready entity=mon.0 client_cookie=ce8268b56d4a8948 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.318+0000 7f6ff7fff700 1 -- 172.21.15.23:0/302288322 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe800b9c0 con 0x7f6ff80faad0 2021-11-19T08:35:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.318+0000 7f6ff7fff700 1 -- 172.21.15.23:0/302288322 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6fe8011400 con 0x7f6ff80faad0 2021-11-19T08:35:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.318+0000 7f6ff7fff700 1 -- 172.21.15.23:0/302288322 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe8010580 con 0x7f6ff80faad0 2021-11-19T08:35:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.319+0000 7f6fff7ce700 1 -- 172.21.15.23:0/302288322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ff80faad0 msgr2=0x7f6ff80faef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:35:57.309 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.319+0000 7f6fff7ce700 1 --2- 172.21.15.23:0/302288322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ff80faad0 0x7f6ff80faef0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f6fe8009960 tx=0x7f6fe80092d0).stop 2021-11-19T08:35:57.309 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.319+0000 7f6fff7ce700 1 -- 172.21.15.23:0/302288322 shutdown_connections 2021-11-19T08:35:57.309 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.319+0000 7f6fff7ce700 1 --2- 172.21.15.23:0/302288322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ff80faad0 0x7f6ff80faef0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:35:57.310 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.319+0000 7f6fff7ce700 1 -- 172.21.15.23:0/302288322 >> 172.21.15.23:0/302288322 conn(0x7f6ff80f6310 msgr2=0x7f6ff80f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:35:57.310 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.319+0000 7f6fff7ce700 1 -- 172.21.15.23:0/302288322 shutdown_connections 2021-11-19T08:35:57.310 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.319+0000 7f6fff7ce700 1 -- 172.21.15.23:0/302288322 wait complete. 2021-11-19T08:35:57.314 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.320+0000 7f6fff7ce700 1 Processor -- start 2021-11-19T08:35:57.314 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.320+0000 7f6fff7ce700 1 -- start start 2021-11-19T08:35:57.314 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.320+0000 7f6fff7ce700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ff80faad0 0x7f6ff810cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:57.315 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.320+0000 7f6fff7ce700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff810d080 con 0x7f6ff80faad0 2021-11-19T08:35:57.315 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.321+0000 7f6ffd56a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ff80faad0 0x7f6ff810cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:35:57.315 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.321+0000 7f6ffd56a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ff80faad0 0x7f6ff810cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46112/0 (socket says 172.21.15.23:46112) 2021-11-19T08:35:57.315 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.321+0000 7f6ffd56a700 1 -- 172.21.15.23:0/3021453703 learned_addr learned my addr 172.21.15.23:0/3021453703 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:35:57.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.321+0000 7f6ffd56a700 1 -- 172.21.15.23:0/3021453703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe8009570 con 0x7f6ff80faad0 2021-11-19T08:35:57.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.321+0000 7f6ffd56a700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ff80faad0 0x7f6ff810cb90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f6fe8011cf0 tx=0x7f6fe80092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:35:57.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.321+0000 7f6ff5ffb700 1 -- 172.21.15.23:0/3021453703 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe80103a0 con 0x7f6ff80faad0 2021-11-19T08:35:57.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.321+0000 7f6ff5ffb700 1 -- 172.21.15.23:0/3021453703 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6fe8010ca0 con 0x7f6ff80faad0 2021-11-19T08:35:57.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.321+0000 7f6fff7ce700 1 -- 172.21.15.23:0/3021453703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff810d280 con 0x7f6ff80faad0 2021-11-19T08:35:57.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.321+0000 7f6ff5ffb700 1 -- 172.21.15.23:0/3021453703 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe801bd00 con 0x7f6ff80faad0 2021-11-19T08:35:57.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.321+0000 7f6fff7ce700 1 -- 172.21.15.23:0/3021453703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff810d690 con 0x7f6ff80faad0 2021-11-19T08:35:57.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.322+0000 7f6ff5ffb700 1 -- 172.21.15.23:0/3021453703 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f6fe800f070 con 0x7f6ff80faad0 2021-11-19T08:35:57.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.322+0000 7f6ff5ffb700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f6fe402fde0 0x7f6fe40322a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:35:57.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.323+0000 7f6ff5ffb700 1 -- 172.21.15.23:0/3021453703 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f6fe8042bc0 con 0x7f6ff80faad0 2021-11-19T08:35:57.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.323+0000 7f6ffcd69700 1 -- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f6fe402fde0 msgr2=0x7f6fe40322a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1352671878 2021-11-19T08:35:57.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.323+0000 7f6ffcd69700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f6fe402fde0 0x7f6fe40322a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-19T08:35:57.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.323+0000 7f6fff7ce700 1 -- 172.21.15.23:0/3021453703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff8059100 con 0x7f6ff80faad0 2021-11-19T08:35:57.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.327+0000 7f6ff5ffb700 1 -- 172.21.15.23:0/3021453703 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6fe801be60 con 0x7f6ff80faad0 2021-11-19T08:35:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.524+0000 7f6ffcd69700 1 -- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f6fe402fde0 msgr2=0x7f6fe40322a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1352671878 2021-11-19T08:35:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.524+0000 7f6ffcd69700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f6fe402fde0 0x7f6fe40322a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-19T08:35:57.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:57 smithi023 conmon[33490]: audit 2021-11-19T08:35:56.289125+0000 mon.smithi023 ( 2021-11-19T08:35:57.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:57 smithi023 conmon[33490]: mon.0) 64 : audit [INF] from='client.? 172.21.15.23:0/2013927429' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2021-11-19T08:35:57.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:35:57 smithi023 conmon[33490]: cluster 2021-11-19T08:35:56.289189+0000 mon.smithi023 (mon.0) 65 : cluster [DBG] mgrmap e9: smithi023.ehlpcl(active, since 13s) 2021-11-19T08:35:57.655 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.668+0000 7f6fff7ce700 1 -- 172.21.15.23:0/3021453703 --> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ff8102590 con 0x7f6fe402fde0 2021-11-19T08:35:57.911 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.925+0000 7f6ffcd69700 1 -- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f6fe402fde0 msgr2=0x7f6fe40322a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1352671878 2021-11-19T08:35:57.911 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:57.925+0000 7f6ffcd69700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f6fe402fde0 0x7f6fe40322a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-19T08:35:58.712 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:58.726+0000 7f6ffcd69700 1 -- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f6fe402fde0 msgr2=0x7f6fe40322a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1352671878 2021-11-19T08:35:58.712 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:35:58.726+0000 7f6ffcd69700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f6fe402fde0 0x7f6fe40322a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-19T08:36:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:00.327+0000 7f6ffcd69700 1 -- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f6fe402fde0 msgr2=0x7f6fe40322a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1352671878 2021-11-19T08:36:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:00.327+0000 7f6ffcd69700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f6fe402fde0 0x7f6fe40322a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-19T08:36:02.282 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:02.295+0000 7f6ff5ffb700 1 -- 172.21.15.23:0/3021453703 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f6fe801b440 con 0x7f6ff80faad0 2021-11-19T08:36:02.283 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:02.295+0000 7f6ff5ffb700 1 -- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f6fe402fde0 msgr2=0x7f6fe40322a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-19T08:36:02.283 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:02.295+0000 7f6ff5ffb700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f6fe402fde0 0x7f6fe40322a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:36:02.533 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:02 smithi023 conmon[33490]: cluster 2021-11-19T08:36:02.243693+0000 mon.smithi023 (mon.0) 66 : cluster [INF] Active manager daemon smithi023.ehlpcl restarted 2021-11-19T08:36:02.533 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:02 smithi023 conmon[33490]: cluster 2021-11-19T08:36:02.244271+0000 mon.smithi023 (mon.0) 67 : cluster [INF] Activating manager daemon smithi023.ehlpcl 2021-11-19T08:36:02.534 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:02 smithi023 conmon[33490]: cluster 2021-11-19T08:36:02.247970+0000 mon.smithi023 (mon.0) 68 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2021-11-19T08:36:03.288 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.301+0000 7f6ff5ffb700 1 -- 172.21.15.23:0/3021453703 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f6fe8019a80 con 0x7f6ff80faad0 2021-11-19T08:36:03.288 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.301+0000 7f6ff5ffb700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f6fe4033c00 0x7f6fe4035ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:03.288 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.301+0000 7f6ff5ffb700 1 -- 172.21.15.23:0/3021453703 --> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ff8102590 con 0x7f6fe4033c00 2021-11-19T08:36:03.289 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.301+0000 7f6ffcd69700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f6fe4033c00 0x7f6fe4035ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:03.289 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.302+0000 7f6ffcd69700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f6fe4033c00 0x7f6fe4035ff0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f6fec006fd0 tx=0x7f6fec006d30).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:03.296 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.310+0000 7f6ff5ffb700 1 -- 172.21.15.23:0/3021453703 <== mgr.14144 v2:172.21.15.23:6800/158654977 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f6ff8102590 con 0x7f6fe4033c00 2021-11-19T08:36:03.296 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Scheduled prometheus update... 2021-11-19T08:36:03.299 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.312+0000 7f6fff7ce700 1 -- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f6fe4033c00 msgr2=0x7f6fe4035ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:03.299 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.312+0000 7f6fff7ce700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f6fe4033c00 0x7f6fe4035ff0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f6fec006fd0 tx=0x7f6fec006d30).stop 2021-11-19T08:36:03.299 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.312+0000 7f6fff7ce700 1 -- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ff80faad0 msgr2=0x7f6ff810cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:03.300 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.312+0000 7f6fff7ce700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ff80faad0 0x7f6ff810cb90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f6fe8011cf0 tx=0x7f6fe80092d0).stop 2021-11-19T08:36:03.300 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.312+0000 7f6fff7ce700 1 -- 172.21.15.23:0/3021453703 shutdown_connections 2021-11-19T08:36:03.300 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.312+0000 7f6fff7ce700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ff80faad0 0x7f6ff810cb90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:03.300 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.312+0000 7f6fff7ce700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f6fe4033c00 0x7f6fe4035ff0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:03.301 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.312+0000 7f6fff7ce700 1 --2- 172.21.15.23:0/3021453703 >> [v2:172.21.15.23:6800/1352671878,v1:172.21.15.23:6801/1352671878] conn(0x7f6fe402fde0 0x7f6fe40322a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:36:03.301 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.312+0000 7f6fff7ce700 1 -- 172.21.15.23:0/3021453703 >> 172.21.15.23:0/3021453703 conn(0x7f6ff80f6310 msgr2=0x7f6ff80ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:03.301 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.313+0000 7f6fff7ce700 1 -- 172.21.15.23:0/3021453703 shutdown_connections 2021-11-19T08:36:03.301 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:03.313+0000 7f6fff7ce700 1 -- 172.21.15.23:0/3021453703 wait complete. 2021-11-19T08:36:03.561 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:03 smithi023 conmon[33490]: cluster 2021-11-19T08:36:02.296809 2021-11-19T08:36:03.561 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:03 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 69 : cluster [DBG] mgrmap e10: smithi023.ehlpcl(active, starting, since 0.0526317s) 2021-11-19T08:36:03.561 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:03 smithi023 conmon[33490]: audit 2021-11-19T08:36:02.299743+0000 mon.smithi023 (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.23:0/1540901876' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi023"}]: dispatch 2021-11-19T08:36:03.562 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:03 smithi023 conmon[33490]: audit 2021-11-19T08:36:02.300463+0000 mon.smithi023 (mon.0) 71 : audit [DBG] from='mgr.14144 172.21.15.23:0/1540901876' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mgr metadata", "who": "smithi023.ehlpcl", "id": "smithi023.ehlpcl"}]: dispatch 2021-11-19T08:36:03.562 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:03 smithi023 conmon[33490]: audit 2021-11-19T08:36:02.301218+0000 mon.smithi023 (mon.0) 72 : audit [DBG] from='mgr.14144 172.21.15.23:0/1540901876' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-19T08:36:03.562 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:03 smithi023 conmon[33490]: audit 2021-11-19T08:36:02.301374+0000 mon.smithi023 (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.23:0/1540901876' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-19T08:36:03.562 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:03 smithi023 conmon[33490]: audit 2021-11-19T08:36:02.301522+0000 mon.smithi023 (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.23:0/1540901876' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-19T08:36:03.563 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:03 smithi023 conmon[33490]: cluster 2021-11-19T08:36:02.303827+0000 mon.smithi023 (mon.0) 75 : cluster [INF] Manager daemon smithi023.ehlpcl is now available 2021-11-19T08:36:03.563 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:03 smithi023 conmon[33490]: audit 2021-11-19T08:36:02.436119+0000 mon.smithi023 (mon.0) 76 : audit [DBG] from='mgr.14144 172.21.15.23:0/1540901876' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:36:03.563 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:03 smithi023 conmon[33490]: audit 2021-11-19T08:36:02.437907+0000 mon.smithi023 (mon.0) 77 : audit [DBG] from='mgr.14144 172.21.15.23:0/1540901876' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:36:03.563 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:03 smithi023 conmon[33490]: audit 2021-11-19T08:36:02.454538+0000 mon.smithi023 (mon.0) 78 : audit [INF] from='mgr.14144 172.21.15.23:0/1540901876' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:36:03.563 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:03 smithi023 conmon[33490]: audit 2021-11-19T08:36:02.456921+0000 mon.smithi023 (mon.0) 79 : audit [INF] from='mgr.14144 172.21.15.23:0/1540901876' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:36:03.791 INFO:teuthology.orchestra.run.smithi023.stderr:Deploying grafana service with default placement... 2021-11-19T08:36:03.791 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2021-11-19T08:36:04.367 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.380+0000 7f954703f700 1 Processor -- start 2021-11-19T08:36:04.368 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.380+0000 7f954703f700 1 -- start start 2021-11-19T08:36:04.368 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.380+0000 7f954703f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95400fc1d0 0x7f95400fc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:04.368 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.380+0000 7f954703f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95400fcae0 con 0x7f95400fc1d0 2021-11-19T08:36:04.369 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.380+0000 7f9544ddb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95400fc1d0 0x7f95400fc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:04.369 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.380+0000 7f9544ddb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95400fc1d0 0x7f95400fc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46156/0 (socket says 172.21.15.23:46156) 2021-11-19T08:36:04.369 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.380+0000 7f9544ddb700 1 -- 172.21.15.23:0/285917025 learned_addr learned my addr 172.21.15.23:0/285917025 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:04.369 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.381+0000 7f9544ddb700 1 -- 172.21.15.23:0/285917025 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95400fcc20 con 0x7f95400fc1d0 2021-11-19T08:36:04.370 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.381+0000 7f9544ddb700 1 --2- 172.21.15.23:0/285917025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95400fc1d0 0x7f95400fc5f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f953400b5f0 tx=0x7f9534006ee0).ready entity=mon.0 client_cookie=4601e848366b4b38 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:04.370 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.381+0000 7f953f7fe700 1 -- 172.21.15.23:0/285917025 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f953400d600 con 0x7f95400fc1d0 2021-11-19T08:36:04.372 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.381+0000 7f953f7fe700 1 -- 172.21.15.23:0/285917025 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f953400dc10 con 0x7f95400fc1d0 2021-11-19T08:36:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.381+0000 7f953f7fe700 1 -- 172.21.15.23:0/285917025 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9534011cd0 con 0x7f95400fc1d0 2021-11-19T08:36:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.382+0000 7f954703f700 1 -- 172.21.15.23:0/285917025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95400fc1d0 msgr2=0x7f95400fc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.382+0000 7f954703f700 1 --2- 172.21.15.23:0/285917025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95400fc1d0 0x7f95400fc5f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f953400b5f0 tx=0x7f9534006ee0).stop 2021-11-19T08:36:04.373 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.382+0000 7f954703f700 1 -- 172.21.15.23:0/285917025 shutdown_connections 2021-11-19T08:36:04.374 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.382+0000 7f954703f700 1 --2- 172.21.15.23:0/285917025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95400fc1d0 0x7f95400fc5f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:04.374 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.382+0000 7f954703f700 1 -- 172.21.15.23:0/285917025 >> 172.21.15.23:0/285917025 conn(0x7f95400f5a80 msgr2=0x7f95400f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:04.374 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.382+0000 7f954703f700 1 -- 172.21.15.23:0/285917025 shutdown_connections 2021-11-19T08:36:04.374 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.382+0000 7f954703f700 1 -- 172.21.15.23:0/285917025 wait complete. 2021-11-19T08:36:04.375 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.382+0000 7f954703f700 1 Processor -- start 2021-11-19T08:36:04.375 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.382+0000 7f954703f700 1 -- start start 2021-11-19T08:36:04.375 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.383+0000 7f954703f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95400fc1d0 0x7f9540111060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:04.375 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.383+0000 7f954703f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9540111550 con 0x7f95400fc1d0 2021-11-19T08:36:04.375 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.383+0000 7f9544ddb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95400fc1d0 0x7f9540111060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:04.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.383+0000 7f9544ddb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95400fc1d0 0x7f9540111060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46158/0 (socket says 172.21.15.23:46158) 2021-11-19T08:36:04.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.383+0000 7f9544ddb700 1 -- 172.21.15.23:0/2234596435 learned_addr learned my addr 172.21.15.23:0/2234596435 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:04.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.383+0000 7f9544ddb700 1 -- 172.21.15.23:0/2234596435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953400b200 con 0x7f95400fc1d0 2021-11-19T08:36:04.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.383+0000 7f9544ddb700 1 --2- 172.21.15.23:0/2234596435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95400fc1d0 0x7f9540111060 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f953400dfb0 tx=0x7f9534006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:04.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.384+0000 7f953d7fa700 1 -- 172.21.15.23:0/2234596435 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9534011df0 con 0x7f95400fc1d0 2021-11-19T08:36:04.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.384+0000 7f953d7fa700 1 -- 172.21.15.23:0/2234596435 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f953401b8c0 con 0x7f95400fc1d0 2021-11-19T08:36:04.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.384+0000 7f954703f700 1 -- 172.21.15.23:0/2234596435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9540111750 con 0x7f95400fc1d0 2021-11-19T08:36:04.378 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.384+0000 7f953d7fa700 1 -- 172.21.15.23:0/2234596435 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f953401a800 con 0x7f95400fc1d0 2021-11-19T08:36:04.378 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.384+0000 7f954703f700 1 -- 172.21.15.23:0/2234596435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9540111b60 con 0x7f95400fc1d0 2021-11-19T08:36:04.378 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.385+0000 7f953d7fa700 1 -- 172.21.15.23:0/2234596435 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f953400f070 con 0x7f95400fc1d0 2021-11-19T08:36:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.385+0000 7f953d7fa700 1 --2- 172.21.15.23:0/2234596435 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f952402fd70 0x7f9524032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.385+0000 7f953d7fa700 1 -- 172.21.15.23:0/2234596435 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f9534040e40 con 0x7f95400fc1d0 2021-11-19T08:36:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.385+0000 7f953ffff700 1 --2- 172.21.15.23:0/2234596435 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f952402fd70 0x7f9524032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.385+0000 7f953ffff700 1 --2- 172.21.15.23:0/2234596435 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f952402fd70 0x7f9524032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f952c0096a0 tx=0x7f952c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.385+0000 7f954703f700 1 -- 172.21.15.23:0/2234596435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9540059100 con 0x7f95400fc1d0 2021-11-19T08:36:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.390+0000 7f953d7fa700 1 -- 172.21.15.23:0/2234596435 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9534016070 con 0x7f95400fc1d0 2021-11-19T08:36:04.575 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:04 smithi023 conmon[33490]: cluster 2021-11-19T08:36:03.303863+0000 2021-11-19T08:36:04.575 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:04 smithi023 conmon[33490]: mon.smithi023 (mon.0) 80 : cluster [DBG] mgrmap e11: smithi023.ehlpcl(active, since 1.05969s) 2021-11-19T08:36:04.575 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:04 smithi023 conmon[33490]: cephadm 2021-11-19T08:36:03.304811+0000 mgr.smithi023.ehlpcl (mgr.14144) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2021-11-19T08:36:04.575 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:04 smithi023 conmon[33490]: audit 2021-11-19T08:36:03.310811+0000 mon.smithi023 (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.23:0/1540901876' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:04.576 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:04 smithi023 conmon[33490]: audit 2021-11-19T08:36:04.134813+0000 mon.smithi023 (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.23:0/1540901876' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:04.731 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.744+0000 7f954703f700 1 -- 172.21.15.23:0/2234596435 --> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f9540100100 con 0x7f952402fd70 2021-11-19T08:36:04.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.751+0000 7f953d7fa700 1 -- 172.21.15.23:0/2234596435 <== mgr.14144 v2:172.21.15.23:6800/158654977 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f9540100100 con 0x7f952402fd70 2021-11-19T08:36:04.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Scheduled grafana update... 2021-11-19T08:36:04.739 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.753+0000 7f954703f700 1 -- 172.21.15.23:0/2234596435 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f952402fd70 msgr2=0x7f9524032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:04.740 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.753+0000 7f954703f700 1 --2- 172.21.15.23:0/2234596435 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f952402fd70 0x7f9524032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f952c0096a0 tx=0x7f952c006b40).stop 2021-11-19T08:36:04.740 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.753+0000 7f954703f700 1 -- 172.21.15.23:0/2234596435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95400fc1d0 msgr2=0x7f9540111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:04.740 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.753+0000 7f954703f700 1 --2- 172.21.15.23:0/2234596435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95400fc1d0 0x7f9540111060 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f953400dfb0 tx=0x7f9534006ee0).stop 2021-11-19T08:36:04.740 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.753+0000 7f954703f700 1 -- 172.21.15.23:0/2234596435 shutdown_connections 2021-11-19T08:36:04.741 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.753+0000 7f954703f700 1 --2- 172.21.15.23:0/2234596435 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f952402fd70 0x7f9524032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:04.741 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.753+0000 7f954703f700 1 --2- 172.21.15.23:0/2234596435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95400fc1d0 0x7f9540111060 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:04.741 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.753+0000 7f954703f700 1 -- 172.21.15.23:0/2234596435 >> 172.21.15.23:0/2234596435 conn(0x7f95400f5a80 msgr2=0x7f95400f7610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:04.741 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.753+0000 7f954703f700 1 -- 172.21.15.23:0/2234596435 shutdown_connections 2021-11-19T08:36:04.742 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:04.753+0000 7f954703f700 1 -- 172.21.15.23:0/2234596435 wait complete. 2021-11-19T08:36:05.631 INFO:teuthology.orchestra.run.smithi023.stderr:Deploying node-exporter service with default placement... 2021-11-19T08:36:05.631 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2021-11-19T08:36:06.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:05 smithi023 conmon[33490]: audit 2021-11-19T08:36:04.746398+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:36:06.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:05 smithi023 conmon[33490]: 14144) 3 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:36:06.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:05 smithi023 conmon[33490]: cephadm 2021-11-19T08:36:04.746950+0000 mgr.smithi023.ehlpcl (mgr.14144) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2021-11-19T08:36:06.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:05 smithi023 conmon[33490]: audit 2021-11-19T08:36:04.751810+0000 mon.smithi023 (mon.0) 83 : audit [INF] from='mgr.14144 172.21.15.23:0/1540901876' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:06.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:05 smithi023 conmon[33490]: cluster 2021-11-19T08:36:05.136260+0000 mon.smithi023 (mon.0) 84 : cluster [DBG] mgrmap e12: smithi023.ehlpcl(active, since 2s) 2021-11-19T08:36:06.883 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.694+0000 7f01337a5700 1 Processor -- start 2021-11-19T08:36:06.883 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.694+0000 7f01337a5700 1 -- start start 2021-11-19T08:36:06.884 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.694+0000 7f01337a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f012c0fc210 0x7f012c0fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:06.884 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.694+0000 7f01337a5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012c0fcb20 con 0x7f012c0fc210 2021-11-19T08:36:06.884 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.695+0000 7f0131541700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f012c0fc210 0x7f012c0fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:06.884 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.695+0000 7f0131541700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f012c0fc210 0x7f012c0fc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46162/0 (socket says 172.21.15.23:46162) 2021-11-19T08:36:06.885 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.695+0000 7f0131541700 1 -- 172.21.15.23:0/3204908029 learned_addr learned my addr 172.21.15.23:0/3204908029 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:06.885 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.695+0000 7f0131541700 1 -- 172.21.15.23:0/3204908029 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c0fcc60 con 0x7f012c0fc210 2021-11-19T08:36:06.885 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.695+0000 7f0131541700 1 --2- 172.21.15.23:0/3204908029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f012c0fc210 0x7f012c0fc630 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f0124009960 tx=0x7f01240092d0).ready entity=mon.0 client_cookie=2cd48be63907d3c4 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:06.885 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.695+0000 7f012bfff700 1 -- 172.21.15.23:0/3204908029 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f012400b9c0 con 0x7f012c0fc210 2021-11-19T08:36:06.886 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.695+0000 7f012bfff700 1 -- 172.21.15.23:0/3204908029 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0124011400 con 0x7f012c0fc210 2021-11-19T08:36:06.886 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.695+0000 7f012bfff700 1 -- 172.21.15.23:0/3204908029 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0124010580 con 0x7f012c0fc210 2021-11-19T08:36:06.886 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.696+0000 7f01337a5700 1 -- 172.21.15.23:0/3204908029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f012c0fc210 msgr2=0x7f012c0fc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:06.887 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.696+0000 7f01337a5700 1 --2- 172.21.15.23:0/3204908029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f012c0fc210 0x7f012c0fc630 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f0124009960 tx=0x7f01240092d0).stop 2021-11-19T08:36:06.887 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.696+0000 7f01337a5700 1 -- 172.21.15.23:0/3204908029 shutdown_connections 2021-11-19T08:36:06.888 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.696+0000 7f01337a5700 1 --2- 172.21.15.23:0/3204908029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f012c0fc210 0x7f012c0fc630 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:06.888 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.696+0000 7f01337a5700 1 -- 172.21.15.23:0/3204908029 >> 172.21.15.23:0/3204908029 conn(0x7f012c0f5a80 msgr2=0x7f012c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:06.888 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.696+0000 7f01337a5700 1 -- 172.21.15.23:0/3204908029 shutdown_connections 2021-11-19T08:36:06.888 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.696+0000 7f01337a5700 1 -- 172.21.15.23:0/3204908029 wait complete. 2021-11-19T08:36:06.889 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.696+0000 7f01337a5700 1 Processor -- start 2021-11-19T08:36:06.889 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.697+0000 7f01337a5700 1 -- start start 2021-11-19T08:36:06.889 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.697+0000 7f01337a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f012c0fc210 0x7f012c10cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:06.889 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.697+0000 7f01337a5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012c10d0c0 con 0x7f012c0fc210 2021-11-19T08:36:06.889 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.697+0000 7f0131541700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f012c0fc210 0x7f012c10cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:06.890 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.697+0000 7f0131541700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f012c0fc210 0x7f012c10cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46164/0 (socket says 172.21.15.23:46164) 2021-11-19T08:36:06.890 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.697+0000 7f0131541700 1 -- 172.21.15.23:0/3320729878 learned_addr learned my addr 172.21.15.23:0/3320729878 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:06.890 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.697+0000 7f0131541700 1 -- 172.21.15.23:0/3320729878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0124009570 con 0x7f012c0fc210 2021-11-19T08:36:06.890 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.697+0000 7f0131541700 1 --2- 172.21.15.23:0/3320729878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f012c0fc210 0x7f012c10cbd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f0124006dc0 tx=0x7f01240092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:06.891 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.697+0000 7f0129ffb700 1 -- 172.21.15.23:0/3320729878 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01240103a0 con 0x7f012c0fc210 2021-11-19T08:36:06.891 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.697+0000 7f0129ffb700 1 -- 172.21.15.23:0/3320729878 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0124010cf0 con 0x7f012c0fc210 2021-11-19T08:36:06.891 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.697+0000 7f01337a5700 1 -- 172.21.15.23:0/3320729878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f012c10d2c0 con 0x7f012c0fc210 2021-11-19T08:36:06.891 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.697+0000 7f0129ffb700 1 -- 172.21.15.23:0/3320729878 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f012401baf0 con 0x7f012c0fc210 2021-11-19T08:36:06.892 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.698+0000 7f01337a5700 1 -- 172.21.15.23:0/3320729878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f012c10d6d0 con 0x7f012c0fc210 2021-11-19T08:36:06.893 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.698+0000 7f0129ffb700 1 -- 172.21.15.23:0/3320729878 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f012400f070 con 0x7f012c0fc210 2021-11-19T08:36:06.893 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.699+0000 7f0129ffb700 1 --2- 172.21.15.23:0/3320729878 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f011802fe60 0x7f0118032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:06.893 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.699+0000 7f0129ffb700 1 -- 172.21.15.23:0/3320729878 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0124041ab0 con 0x7f012c0fc210 2021-11-19T08:36:06.893 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.699+0000 7f01337a5700 1 -- 172.21.15.23:0/3320729878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012c045ad0 con 0x7f012c0fc210 2021-11-19T08:36:06.894 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.699+0000 7f0130d40700 1 --2- 172.21.15.23:0/3320729878 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f011802fe60 0x7f0118032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:06.894 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.699+0000 7f0130d40700 1 --2- 172.21.15.23:0/3320729878 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f011802fe60 0x7f0118032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f01200096a0 tx=0x7f0120006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:06.894 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:06.703+0000 7f0129ffb700 1 -- 172.21.15.23:0/3320729878 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f012401bdb0 con 0x7f012c0fc210 2021-11-19T08:36:07.030 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.044+0000 7f01337a5700 1 -- 172.21.15.23:0/3320729878 --> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f012c106aa0 con 0x7f011802fe60 2021-11-19T08:36:07.034 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.048+0000 7f0129ffb700 1 -- 172.21.15.23:0/3320729878 <== mgr.14144 v2:172.21.15.23:6800/158654977 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f012c106aa0 con 0x7f011802fe60 2021-11-19T08:36:07.034 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2021-11-19T08:36:07.037 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.050+0000 7f01337a5700 1 -- 172.21.15.23:0/3320729878 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f011802fe60 msgr2=0x7f0118032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:07.037 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.050+0000 7f01337a5700 1 --2- 172.21.15.23:0/3320729878 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f011802fe60 0x7f0118032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f01200096a0 tx=0x7f0120006b40).stop 2021-11-19T08:36:07.037 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.050+0000 7f01337a5700 1 -- 172.21.15.23:0/3320729878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f012c0fc210 msgr2=0x7f012c10cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:07.038 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.050+0000 7f01337a5700 1 --2- 172.21.15.23:0/3320729878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f012c0fc210 0x7f012c10cbd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f0124006dc0 tx=0x7f01240092d0).stop 2021-11-19T08:36:07.038 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.050+0000 7f01337a5700 1 -- 172.21.15.23:0/3320729878 shutdown_connections 2021-11-19T08:36:07.038 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.050+0000 7f01337a5700 1 --2- 172.21.15.23:0/3320729878 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f011802fe60 0x7f0118032320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:07.038 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.050+0000 7f01337a5700 1 --2- 172.21.15.23:0/3320729878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f012c0fc210 0x7f012c10cbd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:07.039 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.050+0000 7f01337a5700 1 -- 172.21.15.23:0/3320729878 >> 172.21.15.23:0/3320729878 conn(0x7f012c0f5a80 msgr2=0x7f012c0f75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:07.039 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.050+0000 7f01337a5700 1 -- 172.21.15.23:0/3320729878 shutdown_connections 2021-11-19T08:36:07.039 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.050+0000 7f01337a5700 1 -- 172.21.15.23:0/3320729878 wait complete. 2021-11-19T08:36:07.470 INFO:teuthology.orchestra.run.smithi023.stderr:Deploying alertmanager service with default placement... 2021-11-19T08:36:07.471 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2021-11-19T08:36:07.974 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.982+0000 7ff3dd759700 1 Processor -- start 2021-11-19T08:36:07.974 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.982+0000 7ff3dd759700 1 -- start start 2021-11-19T08:36:07.975 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.982+0000 7ff3dd759700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3d80fa8a0 0x7ff3d80facc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:07.975 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.982+0000 7ff3dd759700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d80fb1b0 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.975 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.982+0000 7ff3d7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3d80fa8a0 0x7ff3d80facc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:07.975 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.983+0000 7ff3d7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3d80fa8a0 0x7ff3d80facc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46168/0 (socket says 172.21.15.23:46168) 2021-11-19T08:36:07.976 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.983+0000 7ff3d7fff700 1 -- 172.21.15.23:0/3779621388 learned_addr learned my addr 172.21.15.23:0/3779621388 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:07.976 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.983+0000 7ff3d7fff700 1 -- 172.21.15.23:0/3779621388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d80fb2f0 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.976 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.983+0000 7ff3d7fff700 1 --2- 172.21.15.23:0/3779621388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3d80fa8a0 0x7ff3d80facc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ff3c800b5f0 tx=0x7ff3c8006ee0).ready entity=mon.0 client_cookie=34b9d76a56f6cc9 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:07.976 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.983+0000 7ff3d6ffd700 1 -- 172.21.15.23:0/3779621388 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3c800d600 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.977 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.983+0000 7ff3d6ffd700 1 -- 172.21.15.23:0/3779621388 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3c800dc10 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.977 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.983+0000 7ff3d6ffd700 1 -- 172.21.15.23:0/3779621388 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3c8011cd0 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.977 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.984+0000 7ff3dd759700 1 -- 172.21.15.23:0/3779621388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3d80fa8a0 msgr2=0x7ff3d80facc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:07.977 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.984+0000 7ff3dd759700 1 --2- 172.21.15.23:0/3779621388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3d80fa8a0 0x7ff3d80facc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ff3c800b5f0 tx=0x7ff3c8006ee0).stop 2021-11-19T08:36:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.984+0000 7ff3dd759700 1 -- 172.21.15.23:0/3779621388 shutdown_connections 2021-11-19T08:36:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.984+0000 7ff3dd759700 1 --2- 172.21.15.23:0/3779621388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3d80fa8a0 0x7ff3d80facc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.984+0000 7ff3dd759700 1 -- 172.21.15.23:0/3779621388 >> 172.21.15.23:0/3779621388 conn(0x7ff3d80f60e0 msgr2=0x7ff3d80f8500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:07.978 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.984+0000 7ff3dd759700 1 -- 172.21.15.23:0/3779621388 shutdown_connections 2021-11-19T08:36:07.979 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.984+0000 7ff3dd759700 1 -- 172.21.15.23:0/3779621388 wait complete. 2021-11-19T08:36:07.979 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.985+0000 7ff3dd759700 1 Processor -- start 2021-11-19T08:36:07.979 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.985+0000 7ff3dd759700 1 -- start start 2021-11-19T08:36:07.979 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.985+0000 7ff3dd759700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3d80fa8a0 0x7ff3d810f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:07.979 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.985+0000 7ff3dd759700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d810b300 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.980 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.985+0000 7ff3d7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3d80fa8a0 0x7ff3d810f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:07.980 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.985+0000 7ff3d7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3d80fa8a0 0x7ff3d810f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46170/0 (socket says 172.21.15.23:46170) 2021-11-19T08:36:07.980 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.985+0000 7ff3d7fff700 1 -- 172.21.15.23:0/689125069 learned_addr learned my addr 172.21.15.23:0/689125069 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:07.980 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.985+0000 7ff3d7fff700 1 -- 172.21.15.23:0/689125069 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c800b200 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.980 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.986+0000 7ff3d7fff700 1 --2- 172.21.15.23:0/689125069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3d80fa8a0 0x7ff3d810f910 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff3c800dfb0 tx=0x7ff3c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:07.981 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.986+0000 7ff3d4ff9700 1 -- 172.21.15.23:0/689125069 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3c8011df0 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.981 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.986+0000 7ff3d4ff9700 1 -- 172.21.15.23:0/689125069 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3c801b8c0 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.981 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.986+0000 7ff3dd759700 1 -- 172.21.15.23:0/689125069 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d806fc50 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.981 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.986+0000 7ff3d4ff9700 1 -- 172.21.15.23:0/689125069 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3c801a8c0 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.982 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.986+0000 7ff3dd759700 1 -- 172.21.15.23:0/689125069 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d810fe00 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.982 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.987+0000 7ff3d4ff9700 1 -- 172.21.15.23:0/689125069 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ff3c800f070 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.982 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.987+0000 7ff3d4ff9700 1 --2- 172.21.15.23:0/689125069 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7ff3b802fe60 0x7ff3b8032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:07.982 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.987+0000 7ff3d4ff9700 1 -- 172.21.15.23:0/689125069 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff3c8041090 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.982 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.987+0000 7ff3d77fe700 1 --2- 172.21.15.23:0/689125069 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7ff3b802fe60 0x7ff3b8032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:07.983 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.987+0000 7ff3d77fe700 1 --2- 172.21.15.23:0/689125069 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7ff3b802fe60 0x7ff3b8032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff3c00096a0 tx=0x7ff3c0006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:07.983 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.988+0000 7ff3dd759700 1 -- 172.21.15.23:0/689125069 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d8045ad0 con 0x7ff3d80fa8a0 2021-11-19T08:36:07.984 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:07.992+0000 7ff3d4ff9700 1 -- 172.21.15.23:0/689125069 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3c8016070 con 0x7ff3d80fa8a0 2021-11-19T08:36:08.317 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:08 smithi023 conmon[33490]: audit 2021-11-19T08:36:07.046088+0000 mgr.smithi023.ehlpcl (mgr.14144) 5 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:36:08.318 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:08 smithi023 conmon[33490]: cephadm 2021-11-19T08:36:07.046810+0000 mgr.smithi023.ehlpcl (mgr.14144) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2021-11-19T08:36:08.318 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:08 smithi023 conmon[33490]: audit 2021-11-19T08:36:07.049054+0000 mon.smithi023 (mon.0) 85 : audit [INF] from='mgr.14144 172.21.15.23:0/1540901876' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:08.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:08.331+0000 7ff3dd759700 1 -- 172.21.15.23:0/689125069 --> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3d80fc790 con 0x7ff3b802fe60 2021-11-19T08:36:08.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:08.336+0000 7ff3d4ff9700 1 -- 172.21.15.23:0/689125069 <== mgr.14144 v2:172.21.15.23:6800/158654977 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7ff3d80fc790 con 0x7ff3b802fe60 2021-11-19T08:36:08.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2021-11-19T08:36:08.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:08.338+0000 7ff3dd759700 1 -- 172.21.15.23:0/689125069 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7ff3b802fe60 msgr2=0x7ff3b8032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:08.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:08.338+0000 7ff3dd759700 1 --2- 172.21.15.23:0/689125069 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7ff3b802fe60 0x7ff3b8032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff3c00096a0 tx=0x7ff3c0006b40).stop 2021-11-19T08:36:08.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:08.338+0000 7ff3dd759700 1 -- 172.21.15.23:0/689125069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3d80fa8a0 msgr2=0x7ff3d810f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:08.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:08.338+0000 7ff3dd759700 1 --2- 172.21.15.23:0/689125069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3d80fa8a0 0x7ff3d810f910 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff3c800dfb0 tx=0x7ff3c8006ee0).stop 2021-11-19T08:36:08.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:08.338+0000 7ff3dd759700 1 -- 172.21.15.23:0/689125069 shutdown_connections 2021-11-19T08:36:08.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:08.338+0000 7ff3dd759700 1 --2- 172.21.15.23:0/689125069 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7ff3b802fe60 0x7ff3b8032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:08.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:08.338+0000 7ff3dd759700 1 --2- 172.21.15.23:0/689125069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3d80fa8a0 0x7ff3d810f910 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:08.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:08.338+0000 7ff3dd759700 1 -- 172.21.15.23:0/689125069 >> 172.21.15.23:0/689125069 conn(0x7ff3d80f60e0 msgr2=0x7ff3d80f7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:08.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:08.338+0000 7ff3dd759700 1 -- 172.21.15.23:0/689125069 shutdown_connections 2021-11-19T08:36:08.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:08.338+0000 7ff3dd759700 1 -- 172.21.15.23:0/689125069 wait complete. 2021-11-19T08:36:08.708 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2021-11-19T08:36:09.218 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.228+0000 7f969b22d700 1 Processor -- start 2021-11-19T08:36:09.218 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.228+0000 7f969b22d700 1 -- start start 2021-11-19T08:36:09.219 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.228+0000 7f969b22d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f969406fbd0 0x7f96940fe810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:09.219 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.228+0000 7f969b22d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96940fed00 con 0x7f969406fbd0 2021-11-19T08:36:09.219 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.229+0000 7f9698fc9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f969406fbd0 0x7f96940fe810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:09.219 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.229+0000 7f9698fc9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f969406fbd0 0x7f96940fe810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46174/0 (socket says 172.21.15.23:46174) 2021-11-19T08:36:09.220 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.229+0000 7f9698fc9700 1 -- 172.21.15.23:0/2875298395 learned_addr learned my addr 172.21.15.23:0/2875298395 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:09.220 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.229+0000 7f9698fc9700 1 -- 172.21.15.23:0/2875298395 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96940fee40 con 0x7f969406fbd0 2021-11-19T08:36:09.220 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.229+0000 7f9698fc9700 1 --2- 172.21.15.23:0/2875298395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f969406fbd0 0x7f96940fe810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f968800b5f0 tx=0x7f9688006ee0).ready entity=mon.0 client_cookie=202f367ff689a3aa server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:09.220 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.229+0000 7f96937fe700 1 -- 172.21.15.23:0/2875298395 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f968800d600 con 0x7f969406fbd0 2021-11-19T08:36:09.221 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.230+0000 7f96937fe700 1 -- 172.21.15.23:0/2875298395 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f968800dc10 con 0x7f969406fbd0 2021-11-19T08:36:09.221 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.230+0000 7f96937fe700 1 -- 172.21.15.23:0/2875298395 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9688011cd0 con 0x7f969406fbd0 2021-11-19T08:36:09.221 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.230+0000 7f969b22d700 1 -- 172.21.15.23:0/2875298395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f969406fbd0 msgr2=0x7f96940fe810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:09.221 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.230+0000 7f969b22d700 1 --2- 172.21.15.23:0/2875298395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f969406fbd0 0x7f96940fe810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f968800b5f0 tx=0x7f9688006ee0).stop 2021-11-19T08:36:09.222 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.230+0000 7f969b22d700 1 -- 172.21.15.23:0/2875298395 shutdown_connections 2021-11-19T08:36:09.222 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.230+0000 7f969b22d700 1 --2- 172.21.15.23:0/2875298395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f969406fbd0 0x7f96940fe810 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:09.222 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.230+0000 7f969b22d700 1 -- 172.21.15.23:0/2875298395 >> 172.21.15.23:0/2875298395 conn(0x7f96940f6270 msgr2=0x7f96940f8690 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:09.222 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.231+0000 7f969b22d700 1 -- 172.21.15.23:0/2875298395 shutdown_connections 2021-11-19T08:36:09.223 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.231+0000 7f969b22d700 1 -- 172.21.15.23:0/2875298395 wait complete. 2021-11-19T08:36:09.223 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.231+0000 7f969b22d700 1 Processor -- start 2021-11-19T08:36:09.223 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.231+0000 7f969b22d700 1 -- start start 2021-11-19T08:36:09.223 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.232+0000 7f969b22d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f969406fbd0 0x7f969410a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:09.223 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.232+0000 7f969b22d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969410ae40 con 0x7f969406fbd0 2021-11-19T08:36:09.224 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.232+0000 7f9698fc9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f969406fbd0 0x7f969410a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:09.224 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.232+0000 7f9698fc9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f969406fbd0 0x7f969410a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46176/0 (socket says 172.21.15.23:46176) 2021-11-19T08:36:09.224 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.232+0000 7f9698fc9700 1 -- 172.21.15.23:0/1955119488 learned_addr learned my addr 172.21.15.23:0/1955119488 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:09.224 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.232+0000 7f9698fc9700 1 -- 172.21.15.23:0/1955119488 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968800b200 con 0x7f969406fbd0 2021-11-19T08:36:09.225 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.232+0000 7f9698fc9700 1 --2- 172.21.15.23:0/1955119488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f969406fbd0 0x7f969410a950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f9688006800 tx=0x7f9688006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:09.226 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.233+0000 7f96917fa700 1 -- 172.21.15.23:0/1955119488 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9688011c90 con 0x7f969406fbd0 2021-11-19T08:36:09.227 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.233+0000 7f96917fa700 1 -- 172.21.15.23:0/1955119488 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f968801b8c0 con 0x7f969406fbd0 2021-11-19T08:36:09.227 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.233+0000 7f969b22d700 1 -- 172.21.15.23:0/1955119488 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969410b040 con 0x7f969406fbd0 2021-11-19T08:36:09.227 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.233+0000 7f969b22d700 1 -- 172.21.15.23:0/1955119488 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969410b4d0 con 0x7f969406fbd0 2021-11-19T08:36:09.227 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.233+0000 7f96917fa700 1 -- 172.21.15.23:0/1955119488 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f968801a880 con 0x7f969406fbd0 2021-11-19T08:36:09.228 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.234+0000 7f96917fa700 1 -- 172.21.15.23:0/1955119488 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f968800f070 con 0x7f969406fbd0 2021-11-19T08:36:09.228 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.234+0000 7f969b22d700 1 -- 172.21.15.23:0/1955119488 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9674004fa0 con 0x7f969406fbd0 2021-11-19T08:36:09.228 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.234+0000 7f96917fa700 1 --2- 172.21.15.23:0/1955119488 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f967802feb0 0x7f9678032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:09.228 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.234+0000 7f96917fa700 1 -- 172.21.15.23:0/1955119488 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f9688040ae0 con 0x7f969406fbd0 2021-11-19T08:36:09.229 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.235+0000 7f9693fff700 1 --2- 172.21.15.23:0/1955119488 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f967802feb0 0x7f9678032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:09.229 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.235+0000 7f9693fff700 1 --2- 172.21.15.23:0/1955119488 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f967802feb0 0x7f9678032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f96800096a0 tx=0x7f9680006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:09.229 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.239+0000 7f96917fa700 1 -- 172.21.15.23:0/1955119488 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f968801ba30 con 0x7f969406fbd0 2021-11-19T08:36:09.513 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:09 smithi023 conmon[33490]: audit 2021-11-19T08:36:08.333420+0000 mgr.smithi023.ehlpcl (mgr.14144) 7 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:36:09.513 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:09 smithi023 conmon[33490]: cephadm 2021-11-19T08:36:08.333958+0000 mgr.smithi023.ehlpcl (mgr.14144) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2021-11-19T08:36:09.514 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:09 smithi023 conmon[33490]: audit 2021-11-19T08:36:08.336871+0000 mon.smithi023 (mon.0) 86 : audit [INF] from='mgr.14144 172.21.15.23:0/1540901876' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:09.514 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.527+0000 7f969b22d700 1 -- 172.21.15.23:0/1955119488 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f9674005670 con 0x7f969406fbd0 2021-11-19T08:36:09.518 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.532+0000 7f96917fa700 1 -- 172.21.15.23:0/1955119488 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f9688016070 con 0x7f969406fbd0 2021-11-19T08:36:09.521 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.534+0000 7f969b22d700 1 -- 172.21.15.23:0/1955119488 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f967802feb0 msgr2=0x7f9678032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:09.521 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.534+0000 7f969b22d700 1 --2- 172.21.15.23:0/1955119488 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f967802feb0 0x7f9678032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f96800096a0 tx=0x7f9680006b40).stop 2021-11-19T08:36:09.521 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.534+0000 7f969b22d700 1 -- 172.21.15.23:0/1955119488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f969406fbd0 msgr2=0x7f969410a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:09.522 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.534+0000 7f969b22d700 1 --2- 172.21.15.23:0/1955119488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f969406fbd0 0x7f969410a950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f9688006800 tx=0x7f9688006ee0).stop 2021-11-19T08:36:09.522 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.534+0000 7f969b22d700 1 -- 172.21.15.23:0/1955119488 shutdown_connections 2021-11-19T08:36:09.522 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.534+0000 7f969b22d700 1 --2- 172.21.15.23:0/1955119488 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f967802feb0 0x7f9678032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:09.522 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.534+0000 7f969b22d700 1 --2- 172.21.15.23:0/1955119488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f969406fbd0 0x7f969410a950 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:09.523 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.534+0000 7f969b22d700 1 -- 172.21.15.23:0/1955119488 >> 172.21.15.23:0/1955119488 conn(0x7f96940f6270 msgr2=0x7f96940f7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:09.523 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.535+0000 7f969b22d700 1 -- 172.21.15.23:0/1955119488 shutdown_connections 2021-11-19T08:36:09.524 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:09.535+0000 7f969b22d700 1 -- 172.21.15.23:0/1955119488 wait complete. 2021-11-19T08:36:09.944 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2021-11-19T08:36:10.395 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.405+0000 7fbc0c927700 1 Processor -- start 2021-11-19T08:36:10.395 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.405+0000 7fbc0c927700 1 -- start start 2021-11-19T08:36:10.395 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.405+0000 7fbc0c927700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc080fd070 0x7fbc080fd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:10.396 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.406+0000 7fbc0c927700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc080fd980 con 0x7fbc080fd070 2021-11-19T08:36:10.396 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.406+0000 7fbc06d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc080fd070 0x7fbc080fd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:10.396 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.406+0000 7fbc06d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc080fd070 0x7fbc080fd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46180/0 (socket says 172.21.15.23:46180) 2021-11-19T08:36:10.397 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.406+0000 7fbc06d9d700 1 -- 172.21.15.23:0/729346512 learned_addr learned my addr 172.21.15.23:0/729346512 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:10.397 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.406+0000 7fbc06d9d700 1 -- 172.21.15.23:0/729346512 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc080fdac0 con 0x7fbc080fd070 2021-11-19T08:36:10.397 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.406+0000 7fbc06d9d700 1 --2- 172.21.15.23:0/729346512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc080fd070 0x7fbc080fd490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fbbf800b5f0 tx=0x7fbbf8006ee0).ready entity=mon.0 client_cookie=53682134e5191479 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:10.397 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.406+0000 7fbc05d9b700 1 -- 172.21.15.23:0/729346512 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbf800d600 con 0x7fbc080fd070 2021-11-19T08:36:10.398 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.406+0000 7fbc05d9b700 1 -- 172.21.15.23:0/729346512 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbf800dc10 con 0x7fbc080fd070 2021-11-19T08:36:10.398 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.406+0000 7fbc05d9b700 1 -- 172.21.15.23:0/729346512 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbf8011cd0 con 0x7fbc080fd070 2021-11-19T08:36:10.398 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.407+0000 7fbc0c927700 1 -- 172.21.15.23:0/729346512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc080fd070 msgr2=0x7fbc080fd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:10.398 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.407+0000 7fbc0c927700 1 --2- 172.21.15.23:0/729346512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc080fd070 0x7fbc080fd490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fbbf800b5f0 tx=0x7fbbf8006ee0).stop 2021-11-19T08:36:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.407+0000 7fbc0c927700 1 -- 172.21.15.23:0/729346512 shutdown_connections 2021-11-19T08:36:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.407+0000 7fbc0c927700 1 --2- 172.21.15.23:0/729346512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc080fd070 0x7fbc080fd490 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.407+0000 7fbc0c927700 1 -- 172.21.15.23:0/729346512 >> 172.21.15.23:0/729346512 conn(0x7fbc080f5870 msgr2=0x7fbc080f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:10.400 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.407+0000 7fbc0c927700 1 -- 172.21.15.23:0/729346512 shutdown_connections 2021-11-19T08:36:10.400 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.407+0000 7fbc0c927700 1 -- 172.21.15.23:0/729346512 wait complete. 2021-11-19T08:36:10.400 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.408+0000 7fbc0c927700 1 Processor -- start 2021-11-19T08:36:10.400 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.408+0000 7fbc0c927700 1 -- start start 2021-11-19T08:36:10.401 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.408+0000 7fbc0c927700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc080fd070 0x7fbc0810a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:10.401 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.408+0000 7fbc0c927700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0810ad70 con 0x7fbc080fd070 2021-11-19T08:36:10.401 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.408+0000 7fbc06d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc080fd070 0x7fbc0810a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:10.402 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.408+0000 7fbc06d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc080fd070 0x7fbc0810a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46182/0 (socket says 172.21.15.23:46182) 2021-11-19T08:36:10.403 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.408+0000 7fbc06d9d700 1 -- 172.21.15.23:0/3083740017 learned_addr learned my addr 172.21.15.23:0/3083740017 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:10.404 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.409+0000 7fbc06d9d700 1 -- 172.21.15.23:0/3083740017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf800b200 con 0x7fbc080fd070 2021-11-19T08:36:10.404 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.409+0000 7fbc06d9d700 1 --2- 172.21.15.23:0/3083740017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc080fd070 0x7fbc0810a880 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fbbf800dfb0 tx=0x7fbbf8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:10.404 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.409+0000 7fbbef7fe700 1 -- 172.21.15.23:0/3083740017 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbf8011df0 con 0x7fbc080fd070 2021-11-19T08:36:10.404 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.409+0000 7fbbef7fe700 1 -- 172.21.15.23:0/3083740017 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbf801b8c0 con 0x7fbc080fd070 2021-11-19T08:36:10.405 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.409+0000 7fbc0c927700 1 -- 172.21.15.23:0/3083740017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0810af70 con 0x7fbc080fd070 2021-11-19T08:36:10.405 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.409+0000 7fbbef7fe700 1 -- 172.21.15.23:0/3083740017 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbf801a7c0 con 0x7fbc080fd070 2021-11-19T08:36:10.405 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.409+0000 7fbc0c927700 1 -- 172.21.15.23:0/3083740017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0810b380 con 0x7fbc080fd070 2021-11-19T08:36:10.405 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.410+0000 7fbbef7fe700 1 -- 172.21.15.23:0/3083740017 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fbbf800f070 con 0x7fbc080fd070 2021-11-19T08:36:10.406 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.410+0000 7fbc0c927700 1 -- 172.21.15.23:0/3083740017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc08045ad0 con 0x7fbc080fd070 2021-11-19T08:36:10.406 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.410+0000 7fbbef7fe700 1 --2- 172.21.15.23:0/3083740017 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbbf402feb0 0x7fbbf4032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:10.406 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.410+0000 7fbbef7fe700 1 -- 172.21.15.23:0/3083740017 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbbf80410d0 con 0x7fbc080fd070 2021-11-19T08:36:10.406 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.411+0000 7fbc0659c700 1 --2- 172.21.15.23:0/3083740017 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbbf402feb0 0x7fbbf4032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:10.407 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.411+0000 7fbc0659c700 1 --2- 172.21.15.23:0/3083740017 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbbf402feb0 0x7fbbf4032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fbbfc0096a0 tx=0x7fbbfc006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:10.407 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.415+0000 7fbbef7fe700 1 -- 172.21.15.23:0/3083740017 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbbf8019d60 con 0x7fbc080fd070 2021-11-19T08:36:10.688 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.702+0000 7fbc0c927700 1 -- 172.21.15.23:0/3083740017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fbc080fa510 con 0x7fbc080fd070 2021-11-19T08:36:10.693 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.707+0000 7fbbef7fe700 1 -- 172.21.15.23:0/3083740017 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7fbbf8016070 con 0x7fbc080fd070 2021-11-19T08:36:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.709+0000 7fbc0c927700 1 -- 172.21.15.23:0/3083740017 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbbf402feb0 msgr2=0x7fbbf4032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.709+0000 7fbc0c927700 1 --2- 172.21.15.23:0/3083740017 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbbf402feb0 0x7fbbf4032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fbbfc0096a0 tx=0x7fbbfc006b40).stop 2021-11-19T08:36:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.709+0000 7fbc0c927700 1 -- 172.21.15.23:0/3083740017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc080fd070 msgr2=0x7fbc0810a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.709+0000 7fbc0c927700 1 --2- 172.21.15.23:0/3083740017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc080fd070 0x7fbc0810a880 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fbbf800dfb0 tx=0x7fbbf8006ee0).stop 2021-11-19T08:36:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.709+0000 7fbc0c927700 1 -- 172.21.15.23:0/3083740017 shutdown_connections 2021-11-19T08:36:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.709+0000 7fbc0c927700 1 --2- 172.21.15.23:0/3083740017 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbbf402feb0 0x7fbbf4032370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.709+0000 7fbc0c927700 1 --2- 172.21.15.23:0/3083740017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc080fd070 0x7fbc0810a880 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.709+0000 7fbc0c927700 1 -- 172.21.15.23:0/3083740017 >> 172.21.15.23:0/3083740017 conn(0x7fbc080f5870 msgr2=0x7fbc080ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:10.698 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.709+0000 7fbc0c927700 1 -- 172.21.15.23:0/3083740017 shutdown_connections 2021-11-19T08:36:10.698 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:10.709+0000 7fbc0c927700 1 -- 172.21.15.23:0/3083740017 wait complete. 2021-11-19T08:36:10.884 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:10 smithi023 conmon[33490]: audit 2021-11-19T08:36:09.533460+0000 mon.smithi023 (mon.0) 87 : audit [INF] from='client.? 172.21.15.23:0/1955119488' entity='client.admin' 2021-11-19T08:36:11.082 INFO:teuthology.orchestra.run.smithi023.stderr:Enabling the dashboard module... 2021-11-19T08:36:11.082 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2021-11-19T08:36:11.553 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.566+0000 7f3d380c5700 1 Processor -- start 2021-11-19T08:36:11.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.566+0000 7f3d380c5700 1 -- start start 2021-11-19T08:36:11.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.566+0000 7f3d380c5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d300faae0 0x7f3d300faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:11.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.566+0000 7f3d380c5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d300fb480 con 0x7f3d300faae0 2021-11-19T08:36:11.554 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.566+0000 7f3d35e61700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d300faae0 0x7f3d300faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:11.555 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.566+0000 7f3d35e61700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d300faae0 0x7f3d300faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46186/0 (socket says 172.21.15.23:46186) 2021-11-19T08:36:11.555 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.566+0000 7f3d35e61700 1 -- 172.21.15.23:0/1699574714 learned_addr learned my addr 172.21.15.23:0/1699574714 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:11.555 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.567+0000 7f3d35e61700 1 -- 172.21.15.23:0/1699574714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d300fb5c0 con 0x7f3d300faae0 2021-11-19T08:36:11.555 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.567+0000 7f3d35e61700 1 --2- 172.21.15.23:0/1699574714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d300faae0 0x7f3d300faf00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f3d2c00b870 tx=0x7f3d2c006f70).ready entity=mon.0 client_cookie=a017d5ef795fd57d server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:11.556 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.567+0000 7f3d34e5f700 1 -- 172.21.15.23:0/1699574714 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d2c00da20 con 0x7f3d300faae0 2021-11-19T08:36:11.556 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.567+0000 7f3d34e5f700 1 -- 172.21.15.23:0/1699574714 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d2c010470 con 0x7f3d300faae0 2021-11-19T08:36:11.556 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.567+0000 7f3d34e5f700 1 -- 172.21.15.23:0/1699574714 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d2c00f5b0 con 0x7f3d300faae0 2021-11-19T08:36:11.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.568+0000 7f3d380c5700 1 -- 172.21.15.23:0/1699574714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d300faae0 msgr2=0x7f3d300faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:11.561 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.568+0000 7f3d380c5700 1 --2- 172.21.15.23:0/1699574714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d300faae0 0x7f3d300faf00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f3d2c00b870 tx=0x7f3d2c006f70).stop 2021-11-19T08:36:11.561 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.568+0000 7f3d380c5700 1 -- 172.21.15.23:0/1699574714 shutdown_connections 2021-11-19T08:36:11.561 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.568+0000 7f3d380c5700 1 --2- 172.21.15.23:0/1699574714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d300faae0 0x7f3d300faf00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:11.561 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.568+0000 7f3d380c5700 1 -- 172.21.15.23:0/1699574714 >> 172.21.15.23:0/1699574714 conn(0x7f3d300f6280 msgr2=0x7f3d300f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:11.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.568+0000 7f3d380c5700 1 -- 172.21.15.23:0/1699574714 shutdown_connections 2021-11-19T08:36:11.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.568+0000 7f3d380c5700 1 -- 172.21.15.23:0/1699574714 wait complete. 2021-11-19T08:36:11.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.568+0000 7f3d380c5700 1 Processor -- start 2021-11-19T08:36:11.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.569+0000 7f3d380c5700 1 -- start start 2021-11-19T08:36:11.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.569+0000 7f3d380c5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d300faae0 0x7f3d3010ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:11.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.569+0000 7f3d380c5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3010d1c0 con 0x7f3d300faae0 2021-11-19T08:36:11.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.569+0000 7f3d35e61700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d300faae0 0x7f3d3010ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:11.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.569+0000 7f3d35e61700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d300faae0 0x7f3d3010ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46188/0 (socket says 172.21.15.23:46188) 2021-11-19T08:36:11.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.569+0000 7f3d35e61700 1 -- 172.21.15.23:0/2265257904 learned_addr learned my addr 172.21.15.23:0/2265257904 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:11.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.569+0000 7f3d35e61700 1 -- 172.21.15.23:0/2265257904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c00b480 con 0x7f3d300faae0 2021-11-19T08:36:11.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.569+0000 7f3d35e61700 1 --2- 172.21.15.23:0/2265257904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d300faae0 0x7f3d3010ccd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f3d2c006b80 tx=0x7f3d2c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:11.565 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.569+0000 7f3d1e7fc700 1 -- 172.21.15.23:0/2265257904 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d2c00f830 con 0x7f3d300faae0 2021-11-19T08:36:11.565 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.569+0000 7f3d380c5700 1 -- 172.21.15.23:0/2265257904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3010d3c0 con 0x7f3d300faae0 2021-11-19T08:36:11.565 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.570+0000 7f3d380c5700 1 -- 172.21.15.23:0/2265257904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3010d7d0 con 0x7f3d300faae0 2021-11-19T08:36:11.565 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.570+0000 7f3d1e7fc700 1 -- 172.21.15.23:0/2265257904 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d2c00fe40 con 0x7f3d300faae0 2021-11-19T08:36:11.566 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.570+0000 7f3d1e7fc700 1 -- 172.21.15.23:0/2265257904 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d2c0223a0 con 0x7f3d300faae0 2021-11-19T08:36:11.566 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.570+0000 7f3d1e7fc700 1 -- 172.21.15.23:0/2265257904 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f3d2c02a070 con 0x7f3d300faae0 2021-11-19T08:36:11.566 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.571+0000 7f3d1e7fc700 1 --2- 172.21.15.23:0/2265257904 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f3d1802feb0 0x7f3d18032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:11.566 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.571+0000 7f3d1e7fc700 1 -- 172.21.15.23:0/2265257904 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f3d2c041ac0 con 0x7f3d300faae0 2021-11-19T08:36:11.566 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.571+0000 7f3d380c5700 1 -- 172.21.15.23:0/2265257904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d10004fa0 con 0x7f3d300faae0 2021-11-19T08:36:11.567 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.571+0000 7f3d35660700 1 --2- 172.21.15.23:0/2265257904 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f3d1802feb0 0x7f3d18032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:11.568 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.571+0000 7f3d35660700 1 --2- 172.21.15.23:0/2265257904 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f3d1802feb0 0x7f3d18032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f3d200096a0 tx=0x7f3d20006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:11.568 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.575+0000 7f3d1e7fc700 1 -- 172.21.15.23:0/2265257904 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d2c014020 con 0x7f3d300faae0 2021-11-19T08:36:11.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:11 smithi023 conmon[33490]: audit 2021-11-19T08:36:10.708473+0000 mon.smithi023 (mon.0) 88 : audit [INF] 2021-11-19T08:36:11.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:11 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/3083740017' entity='client.admin' 2021-11-19T08:36:11.921 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:11.935+0000 7f3d380c5700 1 -- 172.21.15.23:0/2265257904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f3d10005670 con 0x7f3d300faae0 2021-11-19T08:36:12.699 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:12.713+0000 7f3d1e7fc700 1 -- 172.21.15.23:0/2265257904 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f3d2c016030 con 0x7f3d300faae0 2021-11-19T08:36:12.700 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:12.714+0000 7f3d1e7fc700 1 -- 172.21.15.23:0/2265257904 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f3d2c02d9d0 con 0x7f3d300faae0 2021-11-19T08:36:12.704 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:12.717+0000 7f3d380c5700 1 -- 172.21.15.23:0/2265257904 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f3d1802feb0 msgr2=0x7f3d18032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:12.704 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:12.717+0000 7f3d380c5700 1 --2- 172.21.15.23:0/2265257904 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f3d1802feb0 0x7f3d18032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f3d200096a0 tx=0x7f3d20006b40).stop 2021-11-19T08:36:12.704 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:12.717+0000 7f3d380c5700 1 -- 172.21.15.23:0/2265257904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d300faae0 msgr2=0x7f3d3010ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:12.704 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:12.717+0000 7f3d380c5700 1 --2- 172.21.15.23:0/2265257904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d300faae0 0x7f3d3010ccd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f3d2c006b80 tx=0x7f3d2c006f70).stop 2021-11-19T08:36:12.705 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:12.717+0000 7f3d380c5700 1 -- 172.21.15.23:0/2265257904 shutdown_connections 2021-11-19T08:36:12.705 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:12.717+0000 7f3d380c5700 1 --2- 172.21.15.23:0/2265257904 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f3d1802feb0 0x7f3d18032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:12.705 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:12.717+0000 7f3d380c5700 1 --2- 172.21.15.23:0/2265257904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d300faae0 0x7f3d3010ccd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:12.705 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:12.717+0000 7f3d380c5700 1 -- 172.21.15.23:0/2265257904 >> 172.21.15.23:0/2265257904 conn(0x7f3d300f6280 msgr2=0x7f3d300ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:12.706 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:12.717+0000 7f3d380c5700 1 -- 172.21.15.23:0/2265257904 shutdown_connections 2021-11-19T08:36:12.706 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:12.717+0000 7f3d380c5700 1 -- 172.21.15.23:0/2265257904 wait complete. 2021-11-19T08:36:13.076 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:12 smithi023 conmon[33490]: audit 2021-11-19T08:36:11.937007+0000 mon.smithi023 (mon.0) 89 2021-11-19T08:36:13.077 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:12 smithi023 conmon[33490]: : audit [INF] from='client.? 172.21.15.23:0/2265257904' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2021-11-19T08:36:13.121 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2021-11-19T08:36:13.628 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.634+0000 7f662fe43700 1 Processor -- start 2021-11-19T08:36:13.628 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.634+0000 7f662fe43700 1 -- start start 2021-11-19T08:36:13.629 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.635+0000 7f662fe43700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66280faae0 0x7f66280faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:13.629 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.635+0000 7f662fe43700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66280fb480 con 0x7f66280faae0 2021-11-19T08:36:13.630 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.635+0000 7f662dbdf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66280faae0 0x7f66280faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:13.630 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.635+0000 7f662dbdf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66280faae0 0x7f66280faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46202/0 (socket says 172.21.15.23:46202) 2021-11-19T08:36:13.630 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.635+0000 7f662dbdf700 1 -- 172.21.15.23:0/3601650 learned_addr learned my addr 172.21.15.23:0/3601650 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:13.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.635+0000 7f662dbdf700 1 -- 172.21.15.23:0/3601650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66280fb5c0 con 0x7f66280faae0 2021-11-19T08:36:13.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.635+0000 7f662dbdf700 1 --2- 172.21.15.23:0/3601650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66280faae0 0x7f66280faf00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f662400b5f0 tx=0x7f6624006ee0).ready entity=mon.0 client_cookie=f874276441bfa18a server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:13.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.636+0000 7f662cbdd700 1 -- 172.21.15.23:0/3601650 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f662400d600 con 0x7f66280faae0 2021-11-19T08:36:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.636+0000 7f662cbdd700 1 -- 172.21.15.23:0/3601650 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f662400dc10 con 0x7f66280faae0 2021-11-19T08:36:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.636+0000 7f662cbdd700 1 -- 172.21.15.23:0/3601650 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6624011cd0 con 0x7f66280faae0 2021-11-19T08:36:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.636+0000 7f662fe43700 1 -- 172.21.15.23:0/3601650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66280faae0 msgr2=0x7f66280faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.636+0000 7f662fe43700 1 --2- 172.21.15.23:0/3601650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66280faae0 0x7f66280faf00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f662400b5f0 tx=0x7f6624006ee0).stop 2021-11-19T08:36:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.636+0000 7f662fe43700 1 -- 172.21.15.23:0/3601650 shutdown_connections 2021-11-19T08:36:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.636+0000 7f662fe43700 1 --2- 172.21.15.23:0/3601650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66280faae0 0x7f66280faf00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:13.635 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.636+0000 7f662fe43700 1 -- 172.21.15.23:0/3601650 >> 172.21.15.23:0/3601650 conn(0x7f66280f6280 msgr2=0x7f66280f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.637+0000 7f662fe43700 1 -- 172.21.15.23:0/3601650 shutdown_connections 2021-11-19T08:36:13.636 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.637+0000 7f662fe43700 1 -- 172.21.15.23:0/3601650 wait complete. 2021-11-19T08:36:13.637 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.637+0000 7f662fe43700 1 Processor -- start 2021-11-19T08:36:13.637 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.637+0000 7f662fe43700 1 -- start start 2021-11-19T08:36:13.638 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.638+0000 7f662fe43700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66280faae0 0x7f662810aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:13.638 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.638+0000 7f662fe43700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662810af60 con 0x7f66280faae0 2021-11-19T08:36:13.639 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.638+0000 7f662dbdf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66280faae0 0x7f662810aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:13.639 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.638+0000 7f662dbdf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66280faae0 0x7f662810aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46204/0 (socket says 172.21.15.23:46204) 2021-11-19T08:36:13.639 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.638+0000 7f662dbdf700 1 -- 172.21.15.23:0/3409899751 learned_addr learned my addr 172.21.15.23:0/3409899751 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:13.640 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.638+0000 7f662dbdf700 1 -- 172.21.15.23:0/3409899751 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662400b200 con 0x7f66280faae0 2021-11-19T08:36:13.640 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.638+0000 7f662dbdf700 1 --2- 172.21.15.23:0/3409899751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66280faae0 0x7f662810aa70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f662400dfe0 tx=0x7f6624006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:13.641 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.638+0000 7f661a7fc700 1 -- 172.21.15.23:0/3409899751 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6624011df0 con 0x7f66280faae0 2021-11-19T08:36:13.641 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.638+0000 7f662fe43700 1 -- 172.21.15.23:0/3409899751 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662810b160 con 0x7f66280faae0 2021-11-19T08:36:13.642 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.638+0000 7f662fe43700 1 -- 172.21.15.23:0/3409899751 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f662810b570 con 0x7f66280faae0 2021-11-19T08:36:13.642 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.638+0000 7f661a7fc700 1 -- 172.21.15.23:0/3409899751 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f662401b8c0 con 0x7f66280faae0 2021-11-19T08:36:13.643 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.638+0000 7f661a7fc700 1 -- 172.21.15.23:0/3409899751 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f662401a840 con 0x7f66280faae0 2021-11-19T08:36:13.643 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.639+0000 7f661a7fc700 1 -- 172.21.15.23:0/3409899751 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f662400f070 con 0x7f66280faae0 2021-11-19T08:36:13.643 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.640+0000 7f662fe43700 1 -- 172.21.15.23:0/3409899751 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6608004fa0 con 0x7f66280faae0 2021-11-19T08:36:13.644 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.640+0000 7f661a7fc700 1 --2- 172.21.15.23:0/3409899751 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f661002feb0 0x7f6610032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:13.644 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.640+0000 7f662d3de700 1 -- 172.21.15.23:0/3409899751 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f661002feb0 msgr2=0x7f6610032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/158654977 2021-11-19T08:36:13.645 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.640+0000 7f661a7fc700 1 -- 172.21.15.23:0/3409899751 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f6624041000 con 0x7f66280faae0 2021-11-19T08:36:13.646 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.640+0000 7f662d3de700 1 --2- 172.21.15.23:0/3409899751 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f661002feb0 0x7f6610032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-19T08:36:13.646 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.644+0000 7f661a7fc700 1 -- 172.21.15.23:0/3409899751 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6624010bc0 con 0x7f66280faae0 2021-11-19T08:36:13.827 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.841+0000 7f662d3de700 1 -- 172.21.15.23:0/3409899751 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f661002feb0 msgr2=0x7f6610032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/158654977 2021-11-19T08:36:13.827 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:13.841+0000 7f662d3de700 1 --2- 172.21.15.23:0/3409899751 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f661002feb0 0x7f6610032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-19T08:36:13.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:13 smithi023 conmon[33490]: audit 2021-11-19T08 2021-11-19T08:36:13.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:13 smithi023 conmon[33490]: :36:12.715571+0000 mon.smithi023 (mon.0) 90 : audit [INF] from='client.? 172.21.15.23:0/2265257904' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2021-11-19T08:36:13.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:13 smithi023 conmon[33490]: cluster 2021-11-19T08:36:12.715639+0000 mon.smithi023 (mon.0) 91 : cluster [DBG] mgrmap e13: smithi023.ehlpcl(active, since 10s) 2021-11-19T08:36:13.993 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.006+0000 7f662fe43700 1 -- 172.21.15.23:0/3409899751 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f6608005e80 con 0x7f66280faae0 2021-11-19T08:36:14.263 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.006+0000 7f661a7fc700 1 -- 172.21.15.23:0/3409899751 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f6624016070 con 0x7f66280faae0 2021-11-19T08:36:14.264 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2021-11-19T08:36:14.264 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 13, 2021-11-19T08:36:14.265 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "available": true, 2021-11-19T08:36:14.265 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "active_name": "smithi023.ehlpcl", 2021-11-19T08:36:14.266 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_standby": 0 2021-11-19T08:36:14.266 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2021-11-19T08:36:14.270 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.008+0000 7f662fe43700 1 -- 172.21.15.23:0/3409899751 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f661002feb0 msgr2=0x7f6610032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-19T08:36:14.270 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.008+0000 7f662fe43700 1 --2- 172.21.15.23:0/3409899751 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f661002feb0 0x7f6610032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:36:14.271 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.008+0000 7f662fe43700 1 -- 172.21.15.23:0/3409899751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66280faae0 msgr2=0x7f662810aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:14.271 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.008+0000 7f662fe43700 1 --2- 172.21.15.23:0/3409899751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66280faae0 0x7f662810aa70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f662400dfe0 tx=0x7f6624006ee0).stop 2021-11-19T08:36:14.272 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.008+0000 7f662fe43700 1 -- 172.21.15.23:0/3409899751 shutdown_connections 2021-11-19T08:36:14.272 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.008+0000 7f662fe43700 1 --2- 172.21.15.23:0/3409899751 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7f661002feb0 0x7f6610032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:36:14.273 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.008+0000 7f662fe43700 1 --2- 172.21.15.23:0/3409899751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66280faae0 0x7f662810aa70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:14.273 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.008+0000 7f662fe43700 1 -- 172.21.15.23:0/3409899751 >> 172.21.15.23:0/3409899751 conn(0x7f66280f6280 msgr2=0x7f662806f810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:14.273 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.009+0000 7f662fe43700 1 -- 172.21.15.23:0/3409899751 shutdown_connections 2021-11-19T08:36:14.274 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.009+0000 7f662fe43700 1 -- 172.21.15.23:0/3409899751 wait complete. 2021-11-19T08:36:14.359 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for the mgr to restart... 2021-11-19T08:36:14.359 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for mgr epoch 13... 2021-11-19T08:36:14.359 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2021-11-19T08:36:14.861 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.870+0000 7fbc74e5e700 1 Processor -- start 2021-11-19T08:36:14.861 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.870+0000 7fbc74e5e700 1 -- start start 2021-11-19T08:36:14.862 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.870+0000 7fbc74e5e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc7006fbf0 0x7fbc700fe640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:14.863 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.870+0000 7fbc74e5e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc700feb30 con 0x7fbc7006fbf0 2021-11-19T08:36:14.863 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.871+0000 7fbc6f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc7006fbf0 0x7fbc700fe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:14.863 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.871+0000 7fbc6f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc7006fbf0 0x7fbc700fe640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46212/0 (socket says 172.21.15.23:46212) 2021-11-19T08:36:14.864 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.871+0000 7fbc6f7fe700 1 -- 172.21.15.23:0/2553348227 learned_addr learned my addr 172.21.15.23:0/2553348227 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:14.864 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.871+0000 7fbc6f7fe700 1 -- 172.21.15.23:0/2553348227 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc700fec70 con 0x7fbc7006fbf0 2021-11-19T08:36:14.864 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.871+0000 7fbc6f7fe700 1 --2- 172.21.15.23:0/2553348227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc7006fbf0 0x7fbc700fe640 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbc5800b5f0 tx=0x7fbc58006ee0).ready entity=mon.0 client_cookie=becefa2311d749ad server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:14.864 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.871+0000 7fbc6e7fc700 1 -- 172.21.15.23:0/2553348227 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc5800d600 con 0x7fbc7006fbf0 2021-11-19T08:36:14.865 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.871+0000 7fbc6e7fc700 1 -- 172.21.15.23:0/2553348227 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc5800dc10 con 0x7fbc7006fbf0 2021-11-19T08:36:14.865 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.871+0000 7fbc6e7fc700 1 -- 172.21.15.23:0/2553348227 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc58011cd0 con 0x7fbc7006fbf0 2021-11-19T08:36:14.865 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.872+0000 7fbc74e5e700 1 -- 172.21.15.23:0/2553348227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc7006fbf0 msgr2=0x7fbc700fe640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:14.866 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.872+0000 7fbc74e5e700 1 --2- 172.21.15.23:0/2553348227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc7006fbf0 0x7fbc700fe640 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fbc5800b5f0 tx=0x7fbc58006ee0).stop 2021-11-19T08:36:14.866 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.872+0000 7fbc74e5e700 1 -- 172.21.15.23:0/2553348227 shutdown_connections 2021-11-19T08:36:14.866 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.872+0000 7fbc74e5e700 1 --2- 172.21.15.23:0/2553348227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc7006fbf0 0x7fbc700fe640 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:14.866 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.872+0000 7fbc74e5e700 1 -- 172.21.15.23:0/2553348227 >> 172.21.15.23:0/2553348227 conn(0x7fbc700f6000 msgr2=0x7fbc700f8420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:14.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.872+0000 7fbc74e5e700 1 -- 172.21.15.23:0/2553348227 shutdown_connections 2021-11-19T08:36:14.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.872+0000 7fbc74e5e700 1 -- 172.21.15.23:0/2553348227 wait complete. 2021-11-19T08:36:14.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.873+0000 7fbc74e5e700 1 Processor -- start 2021-11-19T08:36:14.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.873+0000 7fbc74e5e700 1 -- start start 2021-11-19T08:36:14.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.873+0000 7fbc74e5e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc7006fbf0 0x7fbc7010a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:14.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.873+0000 7fbc74e5e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7010acc0 con 0x7fbc7006fbf0 2021-11-19T08:36:14.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.873+0000 7fbc6f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc7006fbf0 0x7fbc7010a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:14.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.873+0000 7fbc6f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc7006fbf0 0x7fbc7010a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46214/0 (socket says 172.21.15.23:46214) 2021-11-19T08:36:14.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.873+0000 7fbc6f7fe700 1 -- 172.21.15.23:0/3112164626 learned_addr learned my addr 172.21.15.23:0/3112164626 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:14.869 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.874+0000 7fbc6f7fe700 1 -- 172.21.15.23:0/3112164626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5800b200 con 0x7fbc7006fbf0 2021-11-19T08:36:14.869 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.874+0000 7fbc6f7fe700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc7006fbf0 0x7fbc7010a7d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fbc5800c010 tx=0x7fbc58006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:14.869 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.874+0000 7fbc67fff700 1 -- 172.21.15.23:0/3112164626 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc58011df0 con 0x7fbc7006fbf0 2021-11-19T08:36:14.869 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.874+0000 7fbc67fff700 1 -- 172.21.15.23:0/3112164626 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc5801b8c0 con 0x7fbc7006fbf0 2021-11-19T08:36:14.870 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.874+0000 7fbc67fff700 1 -- 172.21.15.23:0/3112164626 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc5801a830 con 0x7fbc7006fbf0 2021-11-19T08:36:14.870 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.874+0000 7fbc74e5e700 1 -- 172.21.15.23:0/3112164626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc7010aec0 con 0x7fbc7006fbf0 2021-11-19T08:36:14.870 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.874+0000 7fbc74e5e700 1 -- 172.21.15.23:0/3112164626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc7010b350 con 0x7fbc7006fbf0 2021-11-19T08:36:14.870 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.875+0000 7fbc67fff700 1 -- 172.21.15.23:0/3112164626 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fbc5800f070 con 0x7fbc7006fbf0 2021-11-19T08:36:14.872 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.876+0000 7fbc67fff700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbc5002feb0 0x7fbc50032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:14.872 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.876+0000 7fbc67fff700 1 -- 172.21.15.23:0/3112164626 --> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbc50032a30 con 0x7fbc5002feb0 2021-11-19T08:36:14.873 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.876+0000 7fbc67fff700 1 -- 172.21.15.23:0/3112164626 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbc58040f20 con 0x7fbc7006fbf0 2021-11-19T08:36:14.873 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.876+0000 7fbc6effd700 1 -- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbc5002feb0 msgr2=0x7fbc50032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/158654977 2021-11-19T08:36:14.873 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:14.876+0000 7fbc6effd700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbc5002feb0 0x7fbc50032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-19T08:36:15.063 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:14 smithi023 conmon[33490]: audit 2021-11-19T08:36:14.007539+0000 mon.smithi023 (mon.0) 92 2021-11-19T08:36:15.063 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:14 smithi023 conmon[33490]: : audit [DBG] from='client.? 172.21.15.23:0/3409899751' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-11-19T08:36:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:15.077+0000 7fbc6effd700 1 -- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbc5002feb0 msgr2=0x7fbc50032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/158654977 2021-11-19T08:36:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:15.077+0000 7fbc6effd700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbc5002feb0 0x7fbc50032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-19T08:36:15.464 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:15.478+0000 7fbc6effd700 1 -- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbc5002feb0 msgr2=0x7fbc50032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/158654977 2021-11-19T08:36:15.464 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:15.478+0000 7fbc6effd700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbc5002feb0 0x7fbc50032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-19T08:36:16.265 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:16.279+0000 7fbc6effd700 1 -- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbc5002feb0 msgr2=0x7fbc50032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/158654977 2021-11-19T08:36:16.265 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:16.279+0000 7fbc6effd700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbc5002feb0 0x7fbc50032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-19T08:36:17.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:17.880+0000 7fbc6effd700 1 -- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbc5002feb0 msgr2=0x7fbc50032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/158654977 2021-11-19T08:36:17.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:17.880+0000 7fbc6effd700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbc5002feb0 0x7fbc50032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-19T08:36:18.768 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:18.782+0000 7fbc67fff700 1 -- 172.21.15.23:0/3112164626 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fbc5801ba30 con 0x7fbc7006fbf0 2021-11-19T08:36:18.768 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:18.782+0000 7fbc67fff700 1 -- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbc5002feb0 msgr2=0x7fbc50032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-19T08:36:18.769 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:18.782+0000 7fbc67fff700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbc5002feb0 0x7fbc50032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:36:19.030 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:18 smithi023 conmon[33490]: cluster 2021-11-19T08:36:18.730739+0000 mon.smithi023 (mon.0) 93 2021-11-19T08:36:19.030 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:18 smithi023 conmon[33490]: : cluster [INF] Active manager daemon smithi023.ehlpcl restarted 2021-11-19T08:36:19.031 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:18 smithi023 conmon[33490]: cluster 2021-11-19T08:36:18.731351+0000 mon.smithi023 (mon.0) 94 : cluster [INF] Activating manager daemon smithi023.ehlpcl 2021-11-19T08:36:19.031 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:18 smithi023 conmon[33490]: cluster 2021-11-19T08:36:18.734729+0000 mon.smithi023 (mon.0) 95 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-11-19T08:36:19.778 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.790+0000 7fbc67fff700 1 -- 172.21.15.23:0/3112164626 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fbc58042070 con 0x7fbc7006fbf0 2021-11-19T08:36:19.778 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.790+0000 7fbc67fff700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbc50033b40 0x7fbc50035f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:19.778 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.790+0000 7fbc67fff700 1 -- 172.21.15.23:0/3112164626 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbc50032a30 con 0x7fbc50033b40 2021-11-19T08:36:19.779 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.790+0000 7fbc6effd700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbc50033b40 0x7fbc50035f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:19.779 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.791+0000 7fbc6effd700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbc50033b40 0x7fbc50035f30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fbc60006fd0 tx=0x7fbc60006cf0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:19.779 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.792+0000 7fbc67fff700 1 -- 172.21.15.23:0/3112164626 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fbc50032a30 con 0x7fbc50033b40 2021-11-19T08:36:19.788 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.802+0000 7fbc74e5e700 1 -- 172.21.15.23:0/3112164626 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fbc70059100 con 0x7fbc50033b40 2021-11-19T08:36:19.789 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.803+0000 7fbc67fff700 1 -- 172.21.15.23:0/3112164626 <== mgr.14164 v2:172.21.15.23:6800/3049298848 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fbc70059100 con 0x7fbc50033b40 2021-11-19T08:36:19.789 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2021-11-19T08:36:19.789 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2021-11-19T08:36:19.790 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "initialized": true 2021-11-19T08:36:19.791 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2021-11-19T08:36:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.803+0000 7fbc74e5e700 1 -- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbc50033b40 msgr2=0x7fbc50035f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.803+0000 7fbc74e5e700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbc50033b40 0x7fbc50035f30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fbc60006fd0 tx=0x7fbc60006cf0).stop 2021-11-19T08:36:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.803+0000 7fbc74e5e700 1 -- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc7006fbf0 msgr2=0x7fbc7010a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:19.794 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.803+0000 7fbc74e5e700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc7006fbf0 0x7fbc7010a7d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fbc5800c010 tx=0x7fbc58006ee0).stop 2021-11-19T08:36:19.794 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.803+0000 7fbc74e5e700 1 -- 172.21.15.23:0/3112164626 shutdown_connections 2021-11-19T08:36:19.794 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.803+0000 7fbc74e5e700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbc50033b40 0x7fbc50035f30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:19.794 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.803+0000 7fbc74e5e700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc7006fbf0 0x7fbc7010a7d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.803+0000 7fbc74e5e700 1 --2- 172.21.15.23:0/3112164626 >> [v2:172.21.15.23:6800/158654977,v1:172.21.15.23:6801/158654977] conn(0x7fbc5002feb0 0x7fbc50032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:36:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.803+0000 7fbc74e5e700 1 -- 172.21.15.23:0/3112164626 >> 172.21.15.23:0/3112164626 conn(0x7fbc700f6000 msgr2=0x7fbc700f7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.803+0000 7fbc74e5e700 1 -- 172.21.15.23:0/3112164626 shutdown_connections 2021-11-19T08:36:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:19.803+0000 7fbc74e5e700 1 -- 172.21.15.23:0/3112164626 wait complete. 2021-11-19T08:36:20.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:19 smithi023 conmon[33490]: cluster 2021-11-19T08:36:18.783819+0000 mon.smithi023 2021-11-19T08:36:20.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:19 smithi023 conmon[33490]: (mon.0) 96 : cluster [DBG] mgrmap e14: smithi023.ehlpcl(active, starting, since 0.0525511s) 2021-11-19T08:36:20.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:19 smithi023 conmon[33490]: audit 2021-11-19T08:36:18.786660+0000 mon.smithi023 (mon.0) 97 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi023"}]: dispatch 2021-11-19T08:36:20.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:19 smithi023 conmon[33490]: audit 2021-11-19T08:36:18.786871+0000 mon.smithi023 (mon.0) 98 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mgr metadata", "who": "smithi023.ehlpcl", "id": "smithi023.ehlpcl"}]: dispatch 2021-11-19T08:36:20.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:19 smithi023 conmon[33490]: audit 2021-11-19T08:36:18.787070+0000 mon.smithi023 (mon.0) 99 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-11-19T08:36:20.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:19 smithi023 conmon[33490]: audit 2021-11-19T08:36:18.787237+0000 mon.smithi023 (mon.0) 100 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-11-19T08:36:20.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:19 smithi023 conmon[33490]: audit 2021-11-19T08:36:18.787359+0000 mon.smithi023 (mon.0) 101 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-11-19T08:36:20.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:19 smithi023 conmon[33490]: cluster 2021-11-19T08:36:18.801506+0000 mon.smithi023 (mon.0) 102 : cluster [INF] Manager daemon smithi023.ehlpcl is now available 2021-11-19T08:36:20.145 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:19 smithi023 conmon[33490]: audit 2021-11-19T08:36:18.935916+0000 mon.smithi023 (mon.0) 103 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:36:20.145 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:19 smithi023 conmon[33490]: audit 2021-11-19T08:36:18.938561+0000 mon.smithi023 (mon.0) 104 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:36:20.145 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:19 smithi023 conmon[33490]: audit 2021-11-19T08:36:18.966461+0000 mon.smithi023 (mon.0) 105 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:36:20.145 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:19 smithi023 conmon[33490]: audit 2021-11-19T08:36:18.969518+0000 mon.smithi023 (mon.0) 106 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:36:20.206 INFO:teuthology.orchestra.run.smithi023.stderr:mgr epoch 13 is available 2021-11-19T08:36:20.206 INFO:teuthology.orchestra.run.smithi023.stderr:Generating a dashboard self-signed certificate... 2021-11-19T08:36:20.206 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2021-11-19T08:36:20.890 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.900+0000 7f18cd967700 1 Processor -- start 2021-11-19T08:36:20.890 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.900+0000 7f18cd967700 1 -- start start 2021-11-19T08:36:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.901+0000 7f18cd967700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18c806fc70 0x7f18c8100c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.901+0000 7f18cd967700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c80fc680 con 0x7f18c806fc70 2021-11-19T08:36:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.901+0000 7f18c6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18c806fc70 0x7f18c8100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:20.891 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.901+0000 7f18c6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18c806fc70 0x7f18c8100c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46256/0 (socket says 172.21.15.23:46256) 2021-11-19T08:36:20.892 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.901+0000 7f18c6ffd700 1 -- 172.21.15.23:0/227760388 learned_addr learned my addr 172.21.15.23:0/227760388 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:20.892 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.901+0000 7f18c6ffd700 1 -- 172.21.15.23:0/227760388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c81010f0 con 0x7f18c806fc70 2021-11-19T08:36:20.892 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.901+0000 7f18c6ffd700 1 --2- 172.21.15.23:0/227760388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18c806fc70 0x7f18c8100c00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f18b800b5f0 tx=0x7f18b8006ee0).ready entity=mon.0 client_cookie=daddc1853e24b215 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:20.892 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.901+0000 7f18c5ffb700 1 -- 172.21.15.23:0/227760388 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18b800d600 con 0x7f18c806fc70 2021-11-19T08:36:20.893 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.901+0000 7f18c5ffb700 1 -- 172.21.15.23:0/227760388 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18b800dc10 con 0x7f18c806fc70 2021-11-19T08:36:20.893 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.901+0000 7f18c5ffb700 1 -- 172.21.15.23:0/227760388 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18b8011cd0 con 0x7f18c806fc70 2021-11-19T08:36:20.893 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.902+0000 7f18cd967700 1 -- 172.21.15.23:0/227760388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18c806fc70 msgr2=0x7f18c8100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:20.893 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.902+0000 7f18cd967700 1 --2- 172.21.15.23:0/227760388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18c806fc70 0x7f18c8100c00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f18b800b5f0 tx=0x7f18b8006ee0).stop 2021-11-19T08:36:20.893 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.902+0000 7f18cd967700 1 -- 172.21.15.23:0/227760388 shutdown_connections 2021-11-19T08:36:20.894 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.902+0000 7f18cd967700 1 --2- 172.21.15.23:0/227760388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18c806fc70 0x7f18c8100c00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:20.894 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.902+0000 7f18cd967700 1 -- 172.21.15.23:0/227760388 >> 172.21.15.23:0/227760388 conn(0x7f18c80f6280 msgr2=0x7f18c80f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:20.894 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.903+0000 7f18cd967700 1 -- 172.21.15.23:0/227760388 shutdown_connections 2021-11-19T08:36:20.895 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.903+0000 7f18cd967700 1 -- 172.21.15.23:0/227760388 wait complete. 2021-11-19T08:36:20.895 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.903+0000 7f18cd967700 1 Processor -- start 2021-11-19T08:36:20.896 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.903+0000 7f18cd967700 1 -- start start 2021-11-19T08:36:20.896 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.904+0000 7f18cd967700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18c806fc70 0x7f18c810cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:20.896 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.904+0000 7f18cd967700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c810d0e0 con 0x7f18c806fc70 2021-11-19T08:36:20.896 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.904+0000 7f18c6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18c806fc70 0x7f18c810cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:20.897 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.904+0000 7f18c6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18c806fc70 0x7f18c810cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46258/0 (socket says 172.21.15.23:46258) 2021-11-19T08:36:20.897 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.904+0000 7f18c6ffd700 1 -- 172.21.15.23:0/1818478711 learned_addr learned my addr 172.21.15.23:0/1818478711 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:20.897 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.904+0000 7f18c6ffd700 1 -- 172.21.15.23:0/1818478711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b800b200 con 0x7f18c806fc70 2021-11-19T08:36:20.899 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.904+0000 7f18c6ffd700 1 --2- 172.21.15.23:0/1818478711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18c806fc70 0x7f18c810cbf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f18b800b1d0 tx=0x7f18b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:20.899 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.904+0000 7f18af7fe700 1 -- 172.21.15.23:0/1818478711 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18b8011df0 con 0x7f18c806fc70 2021-11-19T08:36:20.899 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.905+0000 7f18cd967700 1 -- 172.21.15.23:0/1818478711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c810d2e0 con 0x7f18c806fc70 2021-11-19T08:36:20.899 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.905+0000 7f18cd967700 1 -- 172.21.15.23:0/1818478711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c810d6f0 con 0x7f18c806fc70 2021-11-19T08:36:20.900 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.905+0000 7f18af7fe700 1 -- 172.21.15.23:0/1818478711 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18b8025910 con 0x7f18c806fc70 2021-11-19T08:36:20.900 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.905+0000 7f18af7fe700 1 -- 172.21.15.23:0/1818478711 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18b801b830 con 0x7f18c806fc70 2021-11-19T08:36:20.900 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.906+0000 7f18ad7fa700 1 -- 172.21.15.23:0/1818478711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a0004fa0 con 0x7f18c806fc70 2021-11-19T08:36:20.900 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.907+0000 7f18af7fe700 1 -- 172.21.15.23:0/1818478711 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f18b800f070 con 0x7f18c806fc70 2021-11-19T08:36:20.900 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.907+0000 7f18af7fe700 1 --2- 172.21.15.23:0/1818478711 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f18b402fdc0 0x7f18b4032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:20.901 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.907+0000 7f18af7fe700 1 -- 172.21.15.23:0/1818478711 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18b8041660 con 0x7f18c806fc70 2021-11-19T08:36:20.901 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.907+0000 7f18c67fc700 1 --2- 172.21.15.23:0/1818478711 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f18b402fdc0 0x7f18b4032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:20.901 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.907+0000 7f18c67fc700 1 --2- 172.21.15.23:0/1818478711 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f18b402fdc0 0x7f18b4032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f18bc0096a0 tx=0x7f18bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:20.901 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:20.911+0000 7f18af7fe700 1 -- 172.21.15.23:0/1818478711 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18b8019d40 con 0x7f18c806fc70 2021-11-19T08:36:21.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:20 smithi023 conmon[33490]: audit 2021-11-19T08:36:19.792693+0000 mgr.smithi023.ehlpcl (mgr.14164) 1 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-11-19T08:36:21.147 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:20 smithi023 conmon[33490]: cluster 2021-11-19T08:36:19.793443+0000 mon.smithi023 (mon.0) 107 : cluster [DBG] mgrmap e15: smithi023.ehlpcl(active, since 1.06218s) 2021-11-19T08:36:21.148 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:20 smithi023 conmon[33490]: audit 2021-11-19T08:36:19.803758+0000 mgr.smithi023.ehlpcl (mgr.14164) 2 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-11-19T08:36:21.148 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:20 smithi023 conmon[33490]: audit 2021-11-19T08:36:20.725731+0000 mon.smithi023 (mon.0) 108 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:21.200 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:21.214+0000 7f18ad7fa700 1 -- 172.21.15.23:0/1818478711 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f18a0000bc0 con 0x7f18b402fdc0 2021-11-19T08:36:21.378 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:21.391+0000 7f18af7fe700 1 -- 172.21.15.23:0/1818478711 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f18a0000bc0 con 0x7f18b402fdc0 2021-11-19T08:36:21.378 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Self-signed certificate created 2021-11-19T08:36:21.380 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:21.393+0000 7f18ad7fa700 1 -- 172.21.15.23:0/1818478711 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f18b402fdc0 msgr2=0x7f18b4032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:21.381 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:21.393+0000 7f18ad7fa700 1 --2- 172.21.15.23:0/1818478711 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f18b402fdc0 0x7f18b4032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f18bc0096a0 tx=0x7f18bc006b40).stop 2021-11-19T08:36:21.381 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:21.393+0000 7f18ad7fa700 1 -- 172.21.15.23:0/1818478711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18c806fc70 msgr2=0x7f18c810cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:21.381 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:21.393+0000 7f18ad7fa700 1 --2- 172.21.15.23:0/1818478711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18c806fc70 0x7f18c810cbf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f18b800b1d0 tx=0x7f18b8006ee0).stop 2021-11-19T08:36:21.382 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:21.393+0000 7f18ad7fa700 1 -- 172.21.15.23:0/1818478711 shutdown_connections 2021-11-19T08:36:21.382 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:21.393+0000 7f18ad7fa700 1 --2- 172.21.15.23:0/1818478711 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f18b402fdc0 0x7f18b4032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:21.382 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:21.393+0000 7f18ad7fa700 1 --2- 172.21.15.23:0/1818478711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18c806fc70 0x7f18c810cbf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:21.382 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:21.393+0000 7f18ad7fa700 1 -- 172.21.15.23:0/1818478711 >> 172.21.15.23:0/1818478711 conn(0x7f18c80f6280 msgr2=0x7f18c80f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:21.383 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:21.393+0000 7f18ad7fa700 1 -- 172.21.15.23:0/1818478711 shutdown_connections 2021-11-19T08:36:21.383 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:21.393+0000 7f18ad7fa700 1 -- 172.21.15.23:0/1818478711 wait complete. 2021-11-19T08:36:22.097 INFO:teuthology.orchestra.run.smithi023.stderr:Creating initial admin user... 2021-11-19T08:36:22.098 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpfqw3lwtx:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2021-11-19T08:36:22.640 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:22 smithi023 conmon[33490]: audit 2021-11-19T08:36:21.215551+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:36:22.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:22 smithi023 conmon[33490]: mgr.14164) 3 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:36:22.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:22 smithi023 conmon[33490]: audit 2021-11-19T08:36:21.383131+0000 mon.smithi023 (mon.0) 109 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:22.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:22 smithi023 conmon[33490]: audit 2021-11-19T08:36:21.392279+0000 mon.smithi023 (mon.0) 110 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:22.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:22 smithi023 conmon[33490]: cluster 2021-11-19T08:36:21.731352+0000 mon.smithi023 (mon.0) 111 : cluster [DBG] mgrmap e16: smithi023.ehlpcl(active, since 3s) 2021-11-19T08:36:22.645 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.654+0000 7f6a86cfe700 1 Processor -- start 2021-11-19T08:36:22.646 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.654+0000 7f6a86cfe700 1 -- start start 2021-11-19T08:36:22.646 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.654+0000 7f6a86cfe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a800fbbc0 0x7f6a800fbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:22.646 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.655+0000 7f6a86cfe700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a800fc4d0 con 0x7f6a800fbbc0 2021-11-19T08:36:22.646 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.655+0000 7f6a84a9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a800fbbc0 0x7f6a800fbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:22.647 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.655+0000 7f6a84a9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a800fbbc0 0x7f6a800fbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46262/0 (socket says 172.21.15.23:46262) 2021-11-19T08:36:22.647 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.655+0000 7f6a84a9a700 1 -- 172.21.15.23:0/2421641346 learned_addr learned my addr 172.21.15.23:0/2421641346 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:22.648 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.655+0000 7f6a84a9a700 1 -- 172.21.15.23:0/2421641346 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a800fc610 con 0x7f6a800fbbc0 2021-11-19T08:36:22.648 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.656+0000 7f6a84a9a700 1 --2- 172.21.15.23:0/2421641346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a800fbbc0 0x7f6a800fbfe0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f6a7400b5f0 tx=0x7f6a74006ee0).ready entity=mon.0 client_cookie=6e704619e0554b0b server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:22.648 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.656+0000 7f6a7f7fe700 1 -- 172.21.15.23:0/2421641346 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a7400d600 con 0x7f6a800fbbc0 2021-11-19T08:36:22.648 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.656+0000 7f6a7f7fe700 1 -- 172.21.15.23:0/2421641346 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a7400dc10 con 0x7f6a800fbbc0 2021-11-19T08:36:22.648 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.656+0000 7f6a7f7fe700 1 -- 172.21.15.23:0/2421641346 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a74011cd0 con 0x7f6a800fbbc0 2021-11-19T08:36:22.649 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.656+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2421641346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a800fbbc0 msgr2=0x7f6a800fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:22.649 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.656+0000 7f6a86cfe700 1 --2- 172.21.15.23:0/2421641346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a800fbbc0 0x7f6a800fbfe0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f6a7400b5f0 tx=0x7f6a74006ee0).stop 2021-11-19T08:36:22.649 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.656+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2421641346 shutdown_connections 2021-11-19T08:36:22.649 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.656+0000 7f6a86cfe700 1 --2- 172.21.15.23:0/2421641346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a800fbbc0 0x7f6a800fbfe0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:22.650 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.656+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2421641346 >> 172.21.15.23:0/2421641346 conn(0x7f6a800f73e0 msgr2=0x7f6a800f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:22.650 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.656+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2421641346 shutdown_connections 2021-11-19T08:36:22.650 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.656+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2421641346 wait complete. 2021-11-19T08:36:22.650 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.657+0000 7f6a86cfe700 1 Processor -- start 2021-11-19T08:36:22.650 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.657+0000 7f6a86cfe700 1 -- start start 2021-11-19T08:36:22.651 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.657+0000 7f6a86cfe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a800fbbc0 0x7f6a80071d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:22.651 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.657+0000 7f6a86cfe700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a800701f0 con 0x7f6a800fbbc0 2021-11-19T08:36:22.651 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.657+0000 7f6a84a9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a800fbbc0 0x7f6a80071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:22.651 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.658+0000 7f6a84a9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a800fbbc0 0x7f6a80071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46264/0 (socket says 172.21.15.23:46264) 2021-11-19T08:36:22.652 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.658+0000 7f6a84a9a700 1 -- 172.21.15.23:0/2616339411 learned_addr learned my addr 172.21.15.23:0/2616339411 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:22.652 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.658+0000 7f6a84a9a700 1 -- 172.21.15.23:0/2616339411 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7400b200 con 0x7f6a800fbbc0 2021-11-19T08:36:22.652 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.658+0000 7f6a84a9a700 1 --2- 172.21.15.23:0/2616339411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a800fbbc0 0x7f6a80071d70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f6a7400dfb0 tx=0x7f6a74006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:22.652 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.658+0000 7f6a7d7fa700 1 -- 172.21.15.23:0/2616339411 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a74011df0 con 0x7f6a800fbbc0 2021-11-19T08:36:22.652 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.658+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2616339411 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a80070390 con 0x7f6a800fbbc0 2021-11-19T08:36:22.653 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.658+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2616339411 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a800707a0 con 0x7f6a800fbbc0 2021-11-19T08:36:22.653 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.658+0000 7f6a7d7fa700 1 -- 172.21.15.23:0/2616339411 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a7401b8c0 con 0x7f6a800fbbc0 2021-11-19T08:36:22.653 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.658+0000 7f6a7d7fa700 1 -- 172.21.15.23:0/2616339411 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a7401a7c0 con 0x7f6a800fbbc0 2021-11-19T08:36:22.653 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.659+0000 7f6a7d7fa700 1 -- 172.21.15.23:0/2616339411 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f6a7400f070 con 0x7f6a800fbbc0 2021-11-19T08:36:22.654 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.659+0000 7f6a7d7fa700 1 --2- 172.21.15.23:0/2616339411 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6a6402feb0 0x7f6a64032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:22.654 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.659+0000 7f6a7d7fa700 1 -- 172.21.15.23:0/2616339411 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a74041170 con 0x7f6a800fbbc0 2021-11-19T08:36:22.654 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.659+0000 7f6a7ffff700 1 --2- 172.21.15.23:0/2616339411 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6a6402feb0 0x7f6a64032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:22.654 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.660+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2616339411 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a80059100 con 0x7f6a800fbbc0 2021-11-19T08:36:22.654 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.660+0000 7f6a7ffff700 1 --2- 172.21.15.23:0/2616339411 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6a6402feb0 0x7f6a64032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6a6c0096a0 tx=0x7f6a6c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:22.655 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.664+0000 7f6a7d7fa700 1 -- 172.21.15.23:0/2616339411 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a74016030 con 0x7f6a800fbbc0 2021-11-19T08:36:22.967 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:22.981+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2616339411 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6a80058f20 con 0x7f6a6402feb0 2021-11-19T08:36:23.218 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:23.232+0000 7f6a7d7fa700 1 -- 172.21.15.23:0/2616339411 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f6a80058f20 con 0x7f6a6402feb0 2021-11-19T08:36:23.219 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$VDcYPw0M8TE1vk57hiojDO6g4Bbp9U8SfW68mKJD0rY3fm8f43/V2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1637310983, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2021-11-19T08:36:23.221 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:23.234+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2616339411 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6a6402feb0 msgr2=0x7f6a64032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:23.221 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:23.234+0000 7f6a86cfe700 1 --2- 172.21.15.23:0/2616339411 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6a6402feb0 0x7f6a64032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6a6c0096a0 tx=0x7f6a6c006b40).stop 2021-11-19T08:36:23.221 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:23.234+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2616339411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a800fbbc0 msgr2=0x7f6a80071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:23.221 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:23.234+0000 7f6a86cfe700 1 --2- 172.21.15.23:0/2616339411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a800fbbc0 0x7f6a80071d70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f6a7400dfb0 tx=0x7f6a74006ee0).stop 2021-11-19T08:36:23.222 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:23.234+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2616339411 shutdown_connections 2021-11-19T08:36:23.222 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:23.234+0000 7f6a86cfe700 1 --2- 172.21.15.23:0/2616339411 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6a6402feb0 0x7f6a64032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:23.222 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:23.234+0000 7f6a86cfe700 1 --2- 172.21.15.23:0/2616339411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a800fbbc0 0x7f6a80071d70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:23.222 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:23.234+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2616339411 >> 172.21.15.23:0/2616339411 conn(0x7f6a800f73e0 msgr2=0x7f6a800f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:23.223 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:23.234+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2616339411 shutdown_connections 2021-11-19T08:36:23.223 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:23.234+0000 7f6a86cfe700 1 -- 172.21.15.23:0/2616339411 wait complete. 2021-11-19T08:36:23.587 INFO:teuthology.orchestra.run.smithi023.stderr:Fetching dashboard port number... 2021-11-19T08:36:23.587 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmpzow9vn7x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpjaatm1ve:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2021-11-19T08:36:24.086 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.099+0000 7f62d2c08700 1 Processor -- start 2021-11-19T08:36:24.087 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.099+0000 7f62d2c08700 1 -- start start 2021-11-19T08:36:24.087 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.099+0000 7f62d2c08700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62cc0faae0 0x7f62cc0faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:24.087 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.099+0000 7f62d2c08700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc0fb480 con 0x7f62cc0faae0 2021-11-19T08:36:24.088 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.099+0000 7f62d09a4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62cc0faae0 0x7f62cc0faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:24.088 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.099+0000 7f62d09a4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62cc0faae0 0x7f62cc0faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46268/0 (socket says 172.21.15.23:46268) 2021-11-19T08:36:24.088 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.099+0000 7f62d09a4700 1 -- 172.21.15.23:0/1087287094 learned_addr learned my addr 172.21.15.23:0/1087287094 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:24.088 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.100+0000 7f62d09a4700 1 -- 172.21.15.23:0/1087287094 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc0fb5c0 con 0x7f62cc0faae0 2021-11-19T08:36:24.088 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.100+0000 7f62d09a4700 1 --2- 172.21.15.23:0/1087287094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62cc0faae0 0x7f62cc0faf00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f62c000b870 tx=0x7f62c0006f70).ready entity=mon.0 client_cookie=86b526a823277a48 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:24.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.100+0000 7f62cb7fe700 1 -- 172.21.15.23:0/1087287094 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62c000da20 con 0x7f62cc0faae0 2021-11-19T08:36:24.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.100+0000 7f62cb7fe700 1 -- 172.21.15.23:0/1087287094 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62c0010470 con 0x7f62cc0faae0 2021-11-19T08:36:24.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.100+0000 7f62cb7fe700 1 -- 172.21.15.23:0/1087287094 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62c000f5b0 con 0x7f62cc0faae0 2021-11-19T08:36:24.092 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.101+0000 7f62d2c08700 1 -- 172.21.15.23:0/1087287094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62cc0faae0 msgr2=0x7f62cc0faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:24.092 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.101+0000 7f62d2c08700 1 --2- 172.21.15.23:0/1087287094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62cc0faae0 0x7f62cc0faf00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f62c000b870 tx=0x7f62c0006f70).stop 2021-11-19T08:36:24.092 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.101+0000 7f62d2c08700 1 -- 172.21.15.23:0/1087287094 shutdown_connections 2021-11-19T08:36:24.093 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.101+0000 7f62d2c08700 1 --2- 172.21.15.23:0/1087287094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62cc0faae0 0x7f62cc0faf00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:24.093 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.101+0000 7f62d2c08700 1 -- 172.21.15.23:0/1087287094 >> 172.21.15.23:0/1087287094 conn(0x7f62cc0f6280 msgr2=0x7f62cc0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:24.093 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.101+0000 7f62d2c08700 1 -- 172.21.15.23:0/1087287094 shutdown_connections 2021-11-19T08:36:24.093 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.101+0000 7f62d2c08700 1 -- 172.21.15.23:0/1087287094 wait complete. 2021-11-19T08:36:24.094 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.102+0000 7f62d2c08700 1 Processor -- start 2021-11-19T08:36:24.094 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.102+0000 7f62d2c08700 1 -- start start 2021-11-19T08:36:24.094 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.102+0000 7f62d2c08700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62cc0faae0 0x7f62cc10ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:24.094 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.102+0000 7f62d2c08700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc10d1c0 con 0x7f62cc0faae0 2021-11-19T08:36:24.094 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.102+0000 7f62d09a4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62cc0faae0 0x7f62cc10ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.102+0000 7f62d09a4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62cc0faae0 0x7f62cc10ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46270/0 (socket says 172.21.15.23:46270) 2021-11-19T08:36:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.102+0000 7f62d09a4700 1 -- 172.21.15.23:0/3583533531 learned_addr learned my addr 172.21.15.23:0/3583533531 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.102+0000 7f62d09a4700 1 -- 172.21.15.23:0/3583533531 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c000b480 con 0x7f62cc0faae0 2021-11-19T08:36:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.103+0000 7f62d09a4700 1 --2- 172.21.15.23:0/3583533531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62cc0faae0 0x7f62cc10ccd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f62c0006b80 tx=0x7f62c0006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:24.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.103+0000 7f62c97fa700 1 -- 172.21.15.23:0/3583533531 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62c000f830 con 0x7f62cc0faae0 2021-11-19T08:36:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.103+0000 7f62d2c08700 1 -- 172.21.15.23:0/3583533531 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62cc10d3c0 con 0x7f62cc0faae0 2021-11-19T08:36:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.103+0000 7f62c97fa700 1 -- 172.21.15.23:0/3583533531 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62c000fe40 con 0x7f62cc0faae0 2021-11-19T08:36:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.103+0000 7f62c97fa700 1 -- 172.21.15.23:0/3583533531 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62c00223a0 con 0x7f62cc0faae0 2021-11-19T08:36:24.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.103+0000 7f62d2c08700 1 -- 172.21.15.23:0/3583533531 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62cc10d7d0 con 0x7f62cc0faae0 2021-11-19T08:36:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.104+0000 7f62d2c08700 1 -- 172.21.15.23:0/3583533531 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62cc045ad0 con 0x7f62cc0faae0 2021-11-19T08:36:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.104+0000 7f62c97fa700 1 -- 172.21.15.23:0/3583533531 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f62c002a070 con 0x7f62cc0faae0 2021-11-19T08:36:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.105+0000 7f62c97fa700 1 --2- 172.21.15.23:0/3583533531 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b002feb0 0x7f62b0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.105+0000 7f62c97fa700 1 -- 172.21.15.23:0/3583533531 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62c0041b70 con 0x7f62cc0faae0 2021-11-19T08:36:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.105+0000 7f62cbfff700 1 --2- 172.21.15.23:0/3583533531 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b002feb0 0x7f62b0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.105+0000 7f62cbfff700 1 --2- 172.21.15.23:0/3583533531 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b002feb0 0x7f62b0032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f62b80096a0 tx=0x7f62b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.113+0000 7f62c97fa700 1 -- 172.21.15.23:0/3583533531 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62c00196a0 con 0x7f62cc0faae0 2021-11-19T08:36:24.385 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.398+0000 7f62d2c08700 1 -- 172.21.15.23:0/3583533531 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f62cc0fc930 con 0x7f62cc0faae0 2021-11-19T08:36:24.385 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.399+0000 7f62c97fa700 1 -- 172.21.15.23:0/3583533531 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f62c0014030 con 0x7f62cc0faae0 2021-11-19T08:36:24.386 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 8443 2021-11-19T08:36:24.387 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.400+0000 7f62d2c08700 1 -- 172.21.15.23:0/3583533531 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b002feb0 msgr2=0x7f62b0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:24.388 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.400+0000 7f62d2c08700 1 --2- 172.21.15.23:0/3583533531 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b002feb0 0x7f62b0032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f62b80096a0 tx=0x7f62b8006b40).stop 2021-11-19T08:36:24.388 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.400+0000 7f62d2c08700 1 -- 172.21.15.23:0/3583533531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62cc0faae0 msgr2=0x7f62cc10ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:24.388 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.400+0000 7f62d2c08700 1 --2- 172.21.15.23:0/3583533531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62cc0faae0 0x7f62cc10ccd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f62c0006b80 tx=0x7f62c0006f70).stop 2021-11-19T08:36:24.388 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.401+0000 7f62d2c08700 1 -- 172.21.15.23:0/3583533531 shutdown_connections 2021-11-19T08:36:24.389 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.401+0000 7f62d2c08700 1 --2- 172.21.15.23:0/3583533531 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b002feb0 0x7f62b0032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:24.389 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.401+0000 7f62d2c08700 1 --2- 172.21.15.23:0/3583533531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62cc0faae0 0x7f62cc10ccd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:24.389 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.401+0000 7f62d2c08700 1 -- 172.21.15.23:0/3583533531 >> 172.21.15.23:0/3583533531 conn(0x7f62cc0f6280 msgr2=0x7f62cc103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:24.389 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.401+0000 7f62d2c08700 1 -- 172.21.15.23:0/3583533531 shutdown_connections 2021-11-19T08:36:24.390 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2021-11-19T08:36:24.401+0000 7f62d2c08700 1 -- 172.21.15.23:0/3583533531 wait complete. 2021-11-19T08:36:24.574 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:24 smithi023 conmon[33490]: audit 2021-11-19T08:36:22.983057+0000 2021-11-19T08:36:24.574 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:24 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 4 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:36:24.574 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:24 smithi023 conmon[33490]: audit 2021-11-19T08:36:23.233244+0000 mon.smithi023 (mon.0) 112 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:24.722 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled firewalld.service 2021-11-19T08:36:24.731 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: disabled 2021-11-19T08:36:24.732 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active firewalld.service 2021-11-19T08:36:24.738 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: inactive 2021-11-19T08:36:24.738 INFO:teuthology.orchestra.run.smithi023.stderr:firewalld.service is not enabled 2021-11-19T08:36:24.739 INFO:teuthology.orchestra.run.smithi023.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2021-11-19T08:36:24.742 INFO:teuthology.orchestra.run.smithi023.stderr:Ceph Dashboard is now available at: 2021-11-19T08:36:24.743 INFO:teuthology.orchestra.run.smithi023.stderr: 2021-11-19T08:36:24.743 INFO:teuthology.orchestra.run.smithi023.stderr: URL: https://smithi023.front.sepia.ceph.com:8443/ 2021-11-19T08:36:24.743 INFO:teuthology.orchestra.run.smithi023.stderr: User: admin 2021-11-19T08:36:24.743 INFO:teuthology.orchestra.run.smithi023.stderr: Password: rq36p92cmb 2021-11-19T08:36:24.744 INFO:teuthology.orchestra.run.smithi023.stderr: 2021-11-19T08:36:24.744 INFO:teuthology.orchestra.run.smithi023.stderr:You can access the Ceph CLI with: 2021-11-19T08:36:24.744 INFO:teuthology.orchestra.run.smithi023.stderr: 2021-11-19T08:36:24.744 INFO:teuthology.orchestra.run.smithi023.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2021-11-19T08:36:24.745 INFO:teuthology.orchestra.run.smithi023.stderr: 2021-11-19T08:36:24.745 INFO:teuthology.orchestra.run.smithi023.stderr:Please consider enabling telemetry to help improve Ceph: 2021-11-19T08:36:24.745 INFO:teuthology.orchestra.run.smithi023.stderr: 2021-11-19T08:36:24.745 INFO:teuthology.orchestra.run.smithi023.stderr: ceph telemetry on 2021-11-19T08:36:24.746 INFO:teuthology.orchestra.run.smithi023.stderr: 2021-11-19T08:36:24.746 INFO:teuthology.orchestra.run.smithi023.stderr:For more information see: 2021-11-19T08:36:24.746 INFO:teuthology.orchestra.run.smithi023.stderr: 2021-11-19T08:36:24.747 INFO:teuthology.orchestra.run.smithi023.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2021-11-19T08:36:24.747 INFO:teuthology.orchestra.run.smithi023.stderr: 2021-11-19T08:36:24.747 INFO:teuthology.orchestra.run.smithi023.stderr:Bootstrap complete. 2021-11-19T08:36:24.747 INFO:teuthology.orchestra.run.smithi023.stderr:Releasing lock 140595876055528 on /run/cephadm/908b41ee-4913-11ec-8c2c-001a4aab830c.lock 2021-11-19T08:36:24.748 INFO:teuthology.orchestra.run.smithi023.stderr:Lock 140595876055528 released on /run/cephadm/908b41ee-4913-11ec-8c2c-001a4aab830c.lock 2021-11-19T08:36:24.776 INFO:tasks.cephadm:Fetching config... 2021-11-19T08:36:24.777 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2021-11-19T08:36:24.777 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2021-11-19T08:36:24.833 INFO:tasks.cephadm:Fetching client.admin keyring... 2021-11-19T08:36:24.834 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2021-11-19T08:36:24.834 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2021-11-19T08:36:24.890 INFO:tasks.cephadm:Fetching mon keyring... 2021-11-19T08:36:24.890 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2021-11-19T08:36:24.891 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/keyring of=/dev/stdout 2021-11-19T08:36:24.959 INFO:tasks.cephadm:Fetching pub ssh key... 2021-11-19T08:36:24.959 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2021-11-19T08:36:24.960 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2021-11-19T08:36:25.016 INFO:tasks.cephadm:Installing pub ssh key for root users... 2021-11-19T08:36:25.017 DEBUG:teuthology.orchestra.run.smithi023:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm+nRJhuzu3Xo9dgX2/W0ZvfH7Pv7PZRW+AIKnun+byC+NyRAiAGA8q4IYKAim7Yj7obvHzxhY8QVBxEJP/1l3ogYzbqrdOQNiX6dAKTSUF1FsbdkXa9ktPlqkj+MM0962nEhsqtpMm5kmzoqUTTuoGYlL3KFMvLLzaP8PzPuZORYcJD/KBCZcRwPDiawFtj9nnof2/7T2nLFhXlP5bhPc2Gfco40k17EQMa4Sp2Cm3U2TYdGk+K/iOnH1uMWUi5K7fJ+y5W+nMpOu9ng2gtl3kAV+Cr0XBGrv2BkfFrboxddrv19KKEMxmmBI70cBteBxGfPexpcUk8A9z9UO3IrWidAtf1CTgbK+3h0Y1zFf4I17xT3iyv6yl2GuyaWYkHm9lQ3Aa7t/vAxQGmkOGj6H9NUYAzxk+z1htKGIOagjkuLhoTeb65haQDcjw8gxTjpqNv3u3zxEIVOzLL0cYfcUMP+lJc5oumfCowI2jdwqPKEh5cqGMRHdHQmXxjtgTT0= ceph-908b41ee-4913-11ec-8c2c-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-11-19T08:36:25.093 INFO:teuthology.orchestra.run.smithi023.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm+nRJhuzu3Xo9dgX2/W0ZvfH7Pv7PZRW+AIKnun+byC+NyRAiAGA8q4IYKAim7Yj7obvHzxhY8QVBxEJP/1l3ogYzbqrdOQNiX6dAKTSUF1FsbdkXa9ktPlqkj+MM0962nEhsqtpMm5kmzoqUTTuoGYlL3KFMvLLzaP8PzPuZORYcJD/KBCZcRwPDiawFtj9nnof2/7T2nLFhXlP5bhPc2Gfco40k17EQMa4Sp2Cm3U2TYdGk+K/iOnH1uMWUi5K7fJ+y5W+nMpOu9ng2gtl3kAV+Cr0XBGrv2BkfFrboxddrv19KKEMxmmBI70cBteBxGfPexpcUk8A9z9UO3IrWidAtf1CTgbK+3h0Y1zFf4I17xT3iyv6yl2GuyaWYkHm9lQ3Aa7t/vAxQGmkOGj6H9NUYAzxk+z1htKGIOagjkuLhoTeb65haQDcjw8gxTjpqNv3u3zxEIVOzLL0cYfcUMP+lJc5oumfCowI2jdwqPKEh5cqGMRHdHQmXxjtgTT0= ceph-908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:36:25.104 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm+nRJhuzu3Xo9dgX2/W0ZvfH7Pv7PZRW+AIKnun+byC+NyRAiAGA8q4IYKAim7Yj7obvHzxhY8QVBxEJP/1l3ogYzbqrdOQNiX6dAKTSUF1FsbdkXa9ktPlqkj+MM0962nEhsqtpMm5kmzoqUTTuoGYlL3KFMvLLzaP8PzPuZORYcJD/KBCZcRwPDiawFtj9nnof2/7T2nLFhXlP5bhPc2Gfco40k17EQMa4Sp2Cm3U2TYdGk+K/iOnH1uMWUi5K7fJ+y5W+nMpOu9ng2gtl3kAV+Cr0XBGrv2BkfFrboxddrv19KKEMxmmBI70cBteBxGfPexpcUk8A9z9UO3IrWidAtf1CTgbK+3h0Y1zFf4I17xT3iyv6yl2GuyaWYkHm9lQ3Aa7t/vAxQGmkOGj6H9NUYAzxk+z1htKGIOagjkuLhoTeb65haQDcjw8gxTjpqNv3u3zxEIVOzLL0cYfcUMP+lJc5oumfCowI2jdwqPKEh5cqGMRHdHQmXxjtgTT0= ceph-908b41ee-4913-11ec-8c2c-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-11-19T08:36:25.146 INFO:teuthology.orchestra.run.smithi134.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCm+nRJhuzu3Xo9dgX2/W0ZvfH7Pv7PZRW+AIKnun+byC+NyRAiAGA8q4IYKAim7Yj7obvHzxhY8QVBxEJP/1l3ogYzbqrdOQNiX6dAKTSUF1FsbdkXa9ktPlqkj+MM0962nEhsqtpMm5kmzoqUTTuoGYlL3KFMvLLzaP8PzPuZORYcJD/KBCZcRwPDiawFtj9nnof2/7T2nLFhXlP5bhPc2Gfco40k17EQMa4Sp2Cm3U2TYdGk+K/iOnH1uMWUi5K7fJ+y5W+nMpOu9ng2gtl3kAV+Cr0XBGrv2BkfFrboxddrv19KKEMxmmBI70cBteBxGfPexpcUk8A9z9UO3IrWidAtf1CTgbK+3h0Y1zFf4I17xT3iyv6yl2GuyaWYkHm9lQ3Aa7t/vAxQGmkOGj6H9NUYAzxk+z1htKGIOagjkuLhoTeb65haQDcjw8gxTjpqNv3u3zxEIVOzLL0cYfcUMP+lJc5oumfCowI2jdwqPKEh5cqGMRHdHQmXxjtgTT0= ceph-908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:36:25.159 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2021-11-19T08:36:25.599 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:25 smithi023 conmon[33490]: audit 2021-11-19T08:36:24.399824+0000 mon.smithi023 (mon.0) 113 : audit [DBG] from='client.? 172.21.15.23:0/3583533531' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2021-11-19T08:36:26.569 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:26 smithi023 conmon[33490]: cluster 2021-11-19T08:36:25.240422+0000 mon.smithi023 (mon 2021-11-19T08:36:26.570 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:26 smithi023 conmon[33490]: .0) 114 : cluster [DBG] mgrmap e17: smithi023.ehlpcl(active, since 6s) 2021-11-19T08:36:26.839 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.851+0000 7f015ec80700 1 -- 172.21.15.23:0/4220696994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01580f5410 msgr2=0x7f01580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:26.839 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.851+0000 7f015ec80700 1 --2- 172.21.15.23:0/4220696994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01580f5410 0x7f01580f5830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f0140009fb0 tx=0x7f0140009b30).stop 2021-11-19T08:36:26.840 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.851+0000 7f015ec80700 1 -- 172.21.15.23:0/4220696994 shutdown_connections 2021-11-19T08:36:26.840 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.851+0000 7f015ec80700 1 --2- 172.21.15.23:0/4220696994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01580f5410 0x7f01580f5830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:26.840 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.851+0000 7f015ec80700 1 -- 172.21.15.23:0/4220696994 >> 172.21.15.23:0/4220696994 conn(0x7f01580f0b70 msgr2=0x7f01580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:26.840 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.851+0000 7f015ec80700 1 -- 172.21.15.23:0/4220696994 shutdown_connections 2021-11-19T08:36:26.841 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.851+0000 7f015ec80700 1 -- 172.21.15.23:0/4220696994 wait complete. 2021-11-19T08:36:26.841 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.851+0000 7f015ec80700 1 Processor -- start 2021-11-19T08:36:26.841 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.851+0000 7f015ec80700 1 -- start start 2021-11-19T08:36:26.841 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.852+0000 7f015ec80700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01580f5410 0x7f0158109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:26.841 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.852+0000 7f015ec80700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0158060220 con 0x7f01580f5410 2021-11-19T08:36:26.842 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.852+0000 7f015ca1c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01580f5410 0x7f0158109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:26.842 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.852+0000 7f015ca1c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01580f5410 0x7f0158109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46280/0 (socket says 172.21.15.23:46280) 2021-11-19T08:36:26.842 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.852+0000 7f015ca1c700 1 -- 172.21.15.23:0/3709390769 learned_addr learned my addr 172.21.15.23:0/3709390769 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:26.842 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.852+0000 7f015ca1c700 1 -- 172.21.15.23:0/3709390769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0140005040 con 0x7f01580f5410 2021-11-19T08:36:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.852+0000 7f015ca1c700 1 --2- 172.21.15.23:0/3709390769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01580f5410 0x7f0158109450 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f0140013510 tx=0x7f014000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.853+0000 7f01557fa700 1 -- 172.21.15.23:0/3709390769 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0140009320 con 0x7f01580f5410 2021-11-19T08:36:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.853+0000 7f01557fa700 1 -- 172.21.15.23:0/3709390769 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0140013f00 con 0x7f01580f5410 2021-11-19T08:36:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.853+0000 7f01557fa700 1 -- 172.21.15.23:0/3709390769 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0140014d90 con 0x7f01580f5410 2021-11-19T08:36:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.853+0000 7f015ec80700 1 -- 172.21.15.23:0/3709390769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0158060420 con 0x7f01580f5410 2021-11-19T08:36:26.844 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.853+0000 7f015ec80700 1 -- 172.21.15.23:0/3709390769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0158109b00 con 0x7f01580f5410 2021-11-19T08:36:26.844 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.853+0000 7f01557fa700 1 -- 172.21.15.23:0/3709390769 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01400263a0 con 0x7f01580f5410 2021-11-19T08:36:26.844 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.854+0000 7f01557fa700 1 --2- 172.21.15.23:0/3709390769 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0144030000 0x7f01440324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:26.844 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.854+0000 7f01557fa700 1 -- 172.21.15.23:0/3709390769 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0140041240 con 0x7f01580f5410 2021-11-19T08:36:26.845 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.854+0000 7f015ec80700 1 -- 172.21.15.23:0/3709390769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0158059100 con 0x7f01580f5410 2021-11-19T08:36:26.845 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.854+0000 7f0157fff700 1 --2- 172.21.15.23:0/3709390769 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0144030000 0x7f01440324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:26.846 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.854+0000 7f0157fff700 1 --2- 172.21.15.23:0/3709390769 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0144030000 0x7f01440324c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f01480096a0 tx=0x7f0148006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:26.846 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:26.858+0000 7f01557fa700 1 -- 172.21.15.23:0/3709390769 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f014001d380 con 0x7f01580f5410 2021-11-19T08:36:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:27.025+0000 7f015ec80700 1 -- 172.21.15.23:0/3709390769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f01580fae60 con 0x7f01580f5410 2021-11-19T08:36:27.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:27.033+0000 7f01557fa700 1 -- 172.21.15.23:0/3709390769 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f014002b030 con 0x7f01580f5410 2021-11-19T08:36:27.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:27.035+0000 7f015ec80700 1 -- 172.21.15.23:0/3709390769 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0144030000 msgr2=0x7f01440324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:27.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:27.035+0000 7f015ec80700 1 --2- 172.21.15.23:0/3709390769 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0144030000 0x7f01440324c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f01480096a0 tx=0x7f0148006b40).stop 2021-11-19T08:36:27.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:27.035+0000 7f015ec80700 1 -- 172.21.15.23:0/3709390769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01580f5410 msgr2=0x7f0158109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:27.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:27.035+0000 7f015ec80700 1 --2- 172.21.15.23:0/3709390769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01580f5410 0x7f0158109450 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f0140013510 tx=0x7f014000b040).stop 2021-11-19T08:36:27.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:27.035+0000 7f015ec80700 1 -- 172.21.15.23:0/3709390769 shutdown_connections 2021-11-19T08:36:27.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:27.035+0000 7f015ec80700 1 --2- 172.21.15.23:0/3709390769 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0144030000 0x7f01440324c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:27.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:27.035+0000 7f015ec80700 1 --2- 172.21.15.23:0/3709390769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01580f5410 0x7f0158109450 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:27.023 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:27.035+0000 7f015ec80700 1 -- 172.21.15.23:0/3709390769 >> 172.21.15.23:0/3709390769 conn(0x7f01580f0b70 msgr2=0x7f01580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:27.023 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:27.035+0000 7f015ec80700 1 -- 172.21.15.23:0/3709390769 shutdown_connections 2021-11-19T08:36:27.023 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:27.035+0000 7f015ec80700 1 -- 172.21.15.23:0/3709390769 wait complete. 2021-11-19T08:36:27.397 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi134 2021-11-19T08:36:27.397 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2021-11-19T08:36:27.398 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/etc/ceph/ceph.conf 2021-11-19T08:36:27.416 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2021-11-19T08:36:27.417 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-19T08:36:27.475 INFO:tasks.cephadm:Adding host smithi134 to orchestrator... 2021-11-19T08:36:27.476 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph orch host add smithi134 2021-11-19T08:36:28.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:28 smithi023 conmon[33490]: audit 2021-11-19T08:36:27.033997+0000 mon.smithi023 (mon.0) 115 : audit [INF] from='client.? 172.21.15.23:0/3709390769' entity='client.admin' 2021-11-19T08:36:28.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:28 smithi023 conmon[33490]: audit 2021-11-19T08:36:27.124714+0000 mon.smithi023 (mon.0) 116 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:28.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:28 smithi023 conmon[33490]: audit 2021-11-19T08:36:27.569194+0000 mon.smithi023 (mon.0) 117 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:28.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:28 smithi023 conmon[33490]: cephadm 2021-11-19T08:36:27.577952+0000 mgr.smithi023.ehlpcl (mgr.14164) 5 : cephadm [INF] Deploying daemon alertmanager.smithi023 on smithi023 2021-11-19T08:36:36.256 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:36 smithi023 conmon[33490]: audit 2021-11-19T08:36:35.097671+0000 mon.smithi023 (mon.0) 118 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:36.256 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:36 smithi023 conmon[33490]: audit 2021-11-19T08:36:35.098712+0000 mon.smithi023 (mon.0) 119 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi023", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-19T08:36:36.257 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:36 smithi023 conmon[33490]: audit 2021-11-19T08:36:35.102086+0000 mon.smithi023 (mon.0) 120 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi023", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2021-11-19T08:36:36.257 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:36 smithi023 conmon[33490]: audit 2021-11-19T08:36:35.102715+0000 mon.smithi023 (mon.0) 121 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:36:36.257 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:36 smithi023 conmon[33490]: cephadm 2021-11-19T08:36:35.103322+0000 mgr.smithi023.ehlpcl (mgr.14164) 6 : cephadm [INF] Deploying daemon crash.smithi023 on smithi023 2021-11-19T08:36:38.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:38 smithi023 conmon[33490]: audit 2021-11-19T08:36:37.286997+0000 mon.smithi023 (mon.0) 122 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:38.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:38 smithi023 conmon[33490]: audit 2021-11-19T08:36:37.472789+0000 mon.smithi023 (mon.0) 123 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:38.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:38 smithi023 conmon[33490]: audit 2021-11-19T08:36:37.479708+0000 mon.smithi023 (mon.0) 124 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:38.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:38 smithi023 conmon[33490]: audit 2021-11-19T08:36:37.480154+0000 mon.smithi023 (mon.0) 125 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-11-19T08:36:38.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:38 smithi023 conmon[33490]: audit 2021-11-19T08:36:37.480656+0000 mgr.smithi023.ehlpcl (mgr.14164) 7 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-11-19T08:36:38.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:38 smithi023 conmon[33490]: audit 2021-11-19T08:36:37.483324+0000 mon.smithi023 (mon.0) 126 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:38.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:38 smithi023 conmon[33490]: cephadm 2021-11-19T08:36:37.486963+0000 mgr.smithi023.ehlpcl (mgr.14164) 8 : cephadm [INF] Deploying daemon grafana.smithi023 on smithi023 2021-11-19T08:36:38.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:38 smithi023 conmon[33490]: 2021-11-19T08:36:40.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:40 smithi023 conmon[33490]: cluster 2021-11-19T08:36:38.787506+0000 mgr.smithi023.ehlpcl 2021-11-19T08:36:40.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:40 smithi023 conmon[33490]: (mgr.14164) 9 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:36:42.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:42 smithi023 conmon[33490]: cluster 2021-11-19T08:36:40.787763+0000 mgr.smithi023.ehlpcl 2021-11-19T08:36:42.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:42 smithi023 conmon[33490]: (mgr.14164) 10 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:36:42.735 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.749+0000 7f71c37fa700 1 -- 172.21.15.134:0/864861144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71bc0f5410 msgr2=0x7f71bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:42.736 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.749+0000 7f71c37fa700 1 --2- 172.21.15.134:0/864861144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71bc0f5410 0x7f71bc0f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f71b8009fb0 tx=0x7f71b8009b30).stop 2021-11-19T08:36:42.736 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.749+0000 7f71c37fa700 1 -- 172.21.15.134:0/864861144 shutdown_connections 2021-11-19T08:36:42.736 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.749+0000 7f71c37fa700 1 --2- 172.21.15.134:0/864861144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71bc0f5410 0x7f71bc0f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:42.736 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.749+0000 7f71c37fa700 1 -- 172.21.15.134:0/864861144 >> 172.21.15.134:0/864861144 conn(0x7f71bc0f0b70 msgr2=0x7f71bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:42.737 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.749+0000 7f71c37fa700 1 -- 172.21.15.134:0/864861144 shutdown_connections 2021-11-19T08:36:42.737 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.749+0000 7f71c37fa700 1 -- 172.21.15.134:0/864861144 wait complete. 2021-11-19T08:36:42.737 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.749+0000 7f71c37fa700 1 Processor -- start 2021-11-19T08:36:42.737 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.750+0000 7f71c37fa700 1 -- start start 2021-11-19T08:36:42.738 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.750+0000 7f71c37fa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71bc0f5410 0x7f71bc109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:42.738 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.750+0000 7f71c37fa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc060220 con 0x7f71bc0f5410 2021-11-19T08:36:42.738 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.750+0000 7f71c1596700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71bc0f5410 0x7f71bc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:42.738 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.750+0000 7f71c1596700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71bc0f5410 0x7f71bc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.134:51312/0 (socket says 172.21.15.134:51312) 2021-11-19T08:36:42.738 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.750+0000 7f71c1596700 1 -- 172.21.15.134:0/3935163058 learned_addr learned my addr 172.21.15.134:0/3935163058 (peer_addr_for_me v2:172.21.15.134:0/0) 2021-11-19T08:36:42.739 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.750+0000 7f71c1596700 1 -- 172.21.15.134:0/3935163058 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b8005040 con 0x7f71bc0f5410 2021-11-19T08:36:42.739 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.750+0000 7f71c1596700 1 --2- 172.21.15.134:0/3935163058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71bc0f5410 0x7f71bc109450 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f71b8013510 tx=0x7f71b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:42.739 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.750+0000 7f71b1ffb700 1 -- 172.21.15.134:0/3935163058 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b8009320 con 0x7f71bc0f5410 2021-11-19T08:36:42.739 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.750+0000 7f71c37fa700 1 -- 172.21.15.134:0/3935163058 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71bc060420 con 0x7f71bc0f5410 2021-11-19T08:36:42.740 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.750+0000 7f71b1ffb700 1 -- 172.21.15.134:0/3935163058 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71b8013f00 con 0x7f71bc0f5410 2021-11-19T08:36:42.740 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.750+0000 7f71b1ffb700 1 -- 172.21.15.134:0/3935163058 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b8014d90 con 0x7f71bc0f5410 2021-11-19T08:36:42.740 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.751+0000 7f71c37fa700 1 -- 172.21.15.134:0/3935163058 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71bc109b00 con 0x7f71bc0f5410 2021-11-19T08:36:42.741 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.751+0000 7f71b1ffb700 1 -- 172.21.15.134:0/3935163058 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71b8014420 con 0x7f71bc0f5410 2021-11-19T08:36:42.742 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.751+0000 7f71c37fa700 1 -- 172.21.15.134:0/3935163058 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71bc059100 con 0x7f71bc0f5410 2021-11-19T08:36:42.742 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.752+0000 7f71b1ffb700 1 --2- 172.21.15.134:0/3935163058 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f71a802ff60 0x7f71a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:42.742 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.752+0000 7f71b1ffb700 1 -- 172.21.15.134:0/3935163058 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f71b80410c0 con 0x7f71bc0f5410 2021-11-19T08:36:42.742 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.752+0000 7f71c0d95700 1 --2- 172.21.15.134:0/3935163058 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f71a802ff60 0x7f71a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:42.742 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.752+0000 7f71c0d95700 1 --2- 172.21.15.134:0/3935163058 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f71a802ff60 0x7f71a8032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f71ac0096a0 tx=0x7f71ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:42.743 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.757+0000 7f71b1ffb700 1 -- 172.21.15.134:0/3935163058 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71b80095f0 con 0x7f71bc0f5410 2021-11-19T08:36:42.913 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:42.927+0000 7f71c37fa700 1 -- 172.21.15.134:0/3935163058 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi134", "target": ["mon-mgr", ""]}) v1 -- 0x7f71bc0f1d70 con 0x7f71a802ff60 2021-11-19T08:36:43.633 INFO:teuthology.orchestra.run.smithi134.stdout:Added host 'smithi134' 2021-11-19T08:36:43.635 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:43.646+0000 7f71b1ffb700 1 -- 172.21.15.134:0/3935163058 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f71bc0f1d70 con 0x7f71a802ff60 2021-11-19T08:36:43.635 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:43.649+0000 7f71c37fa700 1 -- 172.21.15.134:0/3935163058 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f71a802ff60 msgr2=0x7f71a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:43.635 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:43.649+0000 7f71c37fa700 1 --2- 172.21.15.134:0/3935163058 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f71a802ff60 0x7f71a8032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f71ac0096a0 tx=0x7f71ac006b40).stop 2021-11-19T08:36:43.635 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:43.649+0000 7f71c37fa700 1 -- 172.21.15.134:0/3935163058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71bc0f5410 msgr2=0x7f71bc109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:43.636 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:43.649+0000 7f71c37fa700 1 --2- 172.21.15.134:0/3935163058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71bc0f5410 0x7f71bc109450 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f71b8013510 tx=0x7f71b800b040).stop 2021-11-19T08:36:43.636 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:43.649+0000 7f71c37fa700 1 -- 172.21.15.134:0/3935163058 shutdown_connections 2021-11-19T08:36:43.636 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:43.649+0000 7f71c37fa700 1 --2- 172.21.15.134:0/3935163058 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f71a802ff60 0x7f71a8032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:43.636 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:43.649+0000 7f71c37fa700 1 --2- 172.21.15.134:0/3935163058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71bc0f5410 0x7f71bc109450 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:43.637 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:43.649+0000 7f71c37fa700 1 -- 172.21.15.134:0/3935163058 >> 172.21.15.134:0/3935163058 conn(0x7f71bc0f0b70 msgr2=0x7f71bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:43.637 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:43.649+0000 7f71c37fa700 1 -- 172.21.15.134:0/3935163058 shutdown_connections 2021-11-19T08:36:43.637 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:43.649+0000 7f71c37fa700 1 -- 172.21.15.134:0/3935163058 wait complete. 2021-11-19T08:36:44.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph orch host ls --format=json 2021-11-19T08:36:44.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:44 smithi023 conmon[33490]: cluster 2021-11-19T08:36:42.788060+0000 mgr.smithi023.ehlpcl 2021-11-19T08:36:44.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:44 smithi023 conmon[33490]: (mgr.14164) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:36:44.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:44 smithi023 conmon[33490]: audit 2021-11-19T08:36:42.928956+0000 mgr.smithi023.ehlpcl (mgr.14164) 12 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi134", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:36:44.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:44 smithi023 conmon[33490]: audit 2021-11-19T08:36:43.646565+0000 mon.smithi023 (mon.0) 127 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:44.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:44 smithi023 conmon[33490]: cephadm 2021-11-19T 2021-11-19T08:36:44.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:44 smithi023 conmon[33490]: 08:36:43.646948+0000 mgr.smithi023.ehlpcl (mgr.14164) 13 : cephadm 2021-11-19T08:36:44.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:44 smithi023 conmon[33490]: [INF] Added host smithi134 2021-11-19T08:36:46.726 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:46 smithi023 conmon[33490]: cluster 2021-11-19T08:36:44.788333+0000 mgr.smithi023.ehlpcl 2021-11-19T08:36:46.726 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:46 smithi023 conmon[33490]: (mgr.14164) 14 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:36:48.949 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:48 smithi023 conmon[33490]: cluster 2021-11-19T08:36:46.788571+0000 mgr.smithi023.ehlpcl (mgr.14164) 15 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:36:48.950 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:48 smithi023 conmon[33490]: audit 2021-11-19T08:36:48.248235+0000 mon.smithi023 (mon.0) 128 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:49.851 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.862+0000 7f9a192f8700 1 -- 172.21.15.134:0/1272185052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a140f6510 msgr2=0x7f9a140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:49.852 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.862+0000 7f9a192f8700 1 --2- 172.21.15.134:0/1272185052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a140f6510 0x7f9a140f6930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f9a04009fb0 tx=0x7f9a04009b30).stop 2021-11-19T08:36:49.852 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.862+0000 7f9a192f8700 1 -- 172.21.15.134:0/1272185052 shutdown_connections 2021-11-19T08:36:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.862+0000 7f9a192f8700 1 --2- 172.21.15.134:0/1272185052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a140f6510 0x7f9a140f6930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.862+0000 7f9a192f8700 1 -- 172.21.15.134:0/1272185052 >> 172.21.15.134:0/1272185052 conn(0x7f9a140f1c70 msgr2=0x7f9a140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.862+0000 7f9a192f8700 1 -- 172.21.15.134:0/1272185052 shutdown_connections 2021-11-19T08:36:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.862+0000 7f9a192f8700 1 -- 172.21.15.134:0/1272185052 wait complete. 2021-11-19T08:36:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.863+0000 7f9a192f8700 1 Processor -- start 2021-11-19T08:36:49.855 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.863+0000 7f9a192f8700 1 -- start start 2021-11-19T08:36:49.855 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.863+0000 7f9a192f8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a140f6510 0x7f9a14106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:49.855 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.863+0000 7f9a192f8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a14106c20 con 0x7f9a140f6510 2021-11-19T08:36:49.855 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.863+0000 7f9a12ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a140f6510 0x7f9a14106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:49.856 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.863+0000 7f9a12ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a140f6510 0x7f9a14106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.134:51318/0 (socket says 172.21.15.134:51318) 2021-11-19T08:36:49.856 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.863+0000 7f9a12ffd700 1 -- 172.21.15.134:0/708967016 learned_addr learned my addr 172.21.15.134:0/708967016 (peer_addr_for_me v2:172.21.15.134:0/0) 2021-11-19T08:36:49.856 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.864+0000 7f9a12ffd700 1 -- 172.21.15.134:0/708967016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a04005040 con 0x7f9a140f6510 2021-11-19T08:36:49.856 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.864+0000 7f9a12ffd700 1 --2- 172.21.15.134:0/708967016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a140f6510 0x7f9a14106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f9a04013510 tx=0x7f9a0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:49.857 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.864+0000 7f99fb7fe700 1 -- 172.21.15.134:0/708967016 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a04009320 con 0x7f9a140f6510 2021-11-19T08:36:49.857 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.864+0000 7f99fb7fe700 1 -- 172.21.15.134:0/708967016 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a04013f00 con 0x7f9a140f6510 2021-11-19T08:36:49.857 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.864+0000 7f9a192f8700 1 -- 172.21.15.134:0/708967016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a14106e20 con 0x7f9a140f6510 2021-11-19T08:36:49.858 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.864+0000 7f99fb7fe700 1 -- 172.21.15.134:0/708967016 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a04014d90 con 0x7f9a140f6510 2021-11-19T08:36:49.858 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.864+0000 7f9a192f8700 1 -- 172.21.15.134:0/708967016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a141072b0 con 0x7f9a140f6510 2021-11-19T08:36:49.858 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.865+0000 7f99fb7fe700 1 -- 172.21.15.134:0/708967016 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a040263a0 con 0x7f9a140f6510 2021-11-19T08:36:49.858 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.865+0000 7f99fb7fe700 1 --2- 172.21.15.134:0/708967016 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f99fc02ff60 0x7f99fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:49.858 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.865+0000 7f9a192f8700 1 -- 172.21.15.134:0/708967016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a14045ad0 con 0x7f9a140f6510 2021-11-19T08:36:49.859 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.865+0000 7f99fb7fe700 1 -- 172.21.15.134:0/708967016 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a040412f0 con 0x7f9a140f6510 2021-11-19T08:36:49.859 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.865+0000 7f9a127fc700 1 --2- 172.21.15.134:0/708967016 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f99fc02ff60 0x7f99fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:49.859 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.866+0000 7f9a127fc700 1 --2- 172.21.15.134:0/708967016 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f99fc02ff60 0x7f99fc032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f9a080096a0 tx=0x7f9a08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:49.860 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:49.870+0000 7f99fb7fe700 1 -- 172.21.15.134:0/708967016 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a0401cac0 con 0x7f9a140f6510 2021-11-19T08:36:49.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:49 smithi023 conmon[33490]: cephadm 2021-11-19T08:36:48.249938+0000 mgr.smithi023.ehlpcl 2021-11-19T08:36:49.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:49 smithi023 conmon[33490]: (mgr.14164) 16 : cephadm [INF] Deploying daemon node-exporter.smithi023 on smithi023 2021-11-19T08:36:50.025 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:50.040+0000 7f9a192f8700 1 -- 172.21.15.134:0/708967016 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a140f2e70 con 0x7f99fc02ff60 2021-11-19T08:36:50.027 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:50.042+0000 7f99fb7fe700 1 -- 172.21.15.134:0/708967016 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f9a140f2e70 con 0x7f99fc02ff60 2021-11-19T08:36:50.027 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:36:50.028 INFO:teuthology.orchestra.run.smithi134.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "smithi134", "hostname": "smithi134", "labels": [], "status": ""}] 2021-11-19T08:36:50.029 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:50.043+0000 7f9a192f8700 1 -- 172.21.15.134:0/708967016 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f99fc02ff60 msgr2=0x7f99fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:50.029 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:50.043+0000 7f9a192f8700 1 --2- 172.21.15.134:0/708967016 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f99fc02ff60 0x7f99fc032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f9a080096a0 tx=0x7f9a08006b40).stop 2021-11-19T08:36:50.029 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:50.043+0000 7f9a192f8700 1 -- 172.21.15.134:0/708967016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a140f6510 msgr2=0x7f9a14106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:50.029 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:50.043+0000 7f9a192f8700 1 --2- 172.21.15.134:0/708967016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a140f6510 0x7f9a14106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f9a04013510 tx=0x7f9a0400b040).stop 2021-11-19T08:36:50.030 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:50.043+0000 7f9a192f8700 1 -- 172.21.15.134:0/708967016 shutdown_connections 2021-11-19T08:36:50.030 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:50.043+0000 7f9a192f8700 1 --2- 172.21.15.134:0/708967016 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f99fc02ff60 0x7f99fc032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:50.030 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:50.043+0000 7f9a192f8700 1 --2- 172.21.15.134:0/708967016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a140f6510 0x7f9a14106730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:50.030 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:50.043+0000 7f9a192f8700 1 -- 172.21.15.134:0/708967016 >> 172.21.15.134:0/708967016 conn(0x7f9a140f1c70 msgr2=0x7f9a140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:50.031 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:50.043+0000 7f9a192f8700 1 -- 172.21.15.134:0/708967016 shutdown_connections 2021-11-19T08:36:50.031 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:50.043+0000 7f9a192f8700 1 -- 172.21.15.134:0/708967016 wait complete. 2021-11-19T08:36:50.387 INFO:tasks.cephadm:Setting crush tunables to default 2021-11-19T08:36:50.388 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph osd crush tunables default 2021-11-19T08:36:50.735 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:50 smithi023 conmon[33490]: cluster 2021-11-19T08:36:48.788810+0000 2021-11-19T08:36:50.736 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:50 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 17 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:36:52.071 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:51 smithi023 conmon[33490]: audit 2021-11-19T08:36:50.041959+0000 mgr.smithi023.ehlpcl (mgr.14164) 18 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-19T08:36:52.071 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:51 smithi023 conmon[33490]: audit 2021-11-19T08:36:51.550879+0000 mon.smithi023 (mon.0) 129 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:52.359 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.370+0000 7f75016b0700 1 -- 172.21.15.23:0/1221283033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74fc0f4590 msgr2=0x7f74fc0f49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:52.359 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.370+0000 7f75016b0700 1 --2- 172.21.15.23:0/1221283033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74fc0f4590 0x7f74fc0f49b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f74ec009fb0 tx=0x7f74ec009b30).stop 2021-11-19T08:36:52.360 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.370+0000 7f75016b0700 1 -- 172.21.15.23:0/1221283033 shutdown_connections 2021-11-19T08:36:52.360 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.370+0000 7f75016b0700 1 --2- 172.21.15.23:0/1221283033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74fc0f4590 0x7f74fc0f49b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:52.360 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.370+0000 7f75016b0700 1 -- 172.21.15.23:0/1221283033 >> 172.21.15.23:0/1221283033 conn(0x7f74fc0f00b0 msgr2=0x7f74fc0f24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:52.360 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.370+0000 7f75016b0700 1 -- 172.21.15.23:0/1221283033 shutdown_connections 2021-11-19T08:36:52.361 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.370+0000 7f75016b0700 1 -- 172.21.15.23:0/1221283033 wait complete. 2021-11-19T08:36:52.361 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.371+0000 7f75016b0700 1 Processor -- start 2021-11-19T08:36:52.361 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.371+0000 7f75016b0700 1 -- start start 2021-11-19T08:36:52.361 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.371+0000 7f75016b0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74fc0f4590 0x7f74fc106580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:52.362 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.371+0000 7f75016b0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc106a70 con 0x7f74fc0f4590 2021-11-19T08:36:52.362 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.372+0000 7f74faffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74fc0f4590 0x7f74fc106580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:52.362 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.372+0000 7f74faffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74fc0f4590 0x7f74fc106580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46342/0 (socket says 172.21.15.23:46342) 2021-11-19T08:36:52.362 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.372+0000 7f74faffd700 1 -- 172.21.15.23:0/4001332821 learned_addr learned my addr 172.21.15.23:0/4001332821 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:36:52.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.372+0000 7f74faffd700 1 -- 172.21.15.23:0/4001332821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ec005040 con 0x7f74fc0f4590 2021-11-19T08:36:52.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.372+0000 7f74faffd700 1 --2- 172.21.15.23:0/4001332821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74fc0f4590 0x7f74fc106580 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f74ec013510 tx=0x7f74ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:52.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.372+0000 7f74e37fe700 1 -- 172.21.15.23:0/4001332821 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74ec009320 con 0x7f74fc0f4590 2021-11-19T08:36:52.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.372+0000 7f74e37fe700 1 -- 172.21.15.23:0/4001332821 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74ec013f00 con 0x7f74fc0f4590 2021-11-19T08:36:52.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.372+0000 7f75016b0700 1 -- 172.21.15.23:0/4001332821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74fc106c70 con 0x7f74fc0f4590 2021-11-19T08:36:52.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.372+0000 7f74e37fe700 1 -- 172.21.15.23:0/4001332821 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74ec014d90 con 0x7f74fc0f4590 2021-11-19T08:36:52.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.372+0000 7f75016b0700 1 -- 172.21.15.23:0/4001332821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74fc107100 con 0x7f74fc0f4590 2021-11-19T08:36:52.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.373+0000 7f74e37fe700 1 -- 172.21.15.23:0/4001332821 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74ec0263a0 con 0x7f74fc0f4590 2021-11-19T08:36:52.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.373+0000 7f74e37fe700 1 --2- 172.21.15.23:0/4001332821 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f74e402ff60 0x7f74e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:52.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.373+0000 7f75016b0700 1 -- 172.21.15.23:0/4001332821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74fc045ad0 con 0x7f74fc0f4590 2021-11-19T08:36:52.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.373+0000 7f74e37fe700 1 -- 172.21.15.23:0/4001332821 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74ec040ff0 con 0x7f74fc0f4590 2021-11-19T08:36:52.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.373+0000 7f74fa7fc700 1 --2- 172.21.15.23:0/4001332821 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f74e402ff60 0x7f74e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:52.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.374+0000 7f74fa7fc700 1 --2- 172.21.15.23:0/4001332821 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f74e402ff60 0x7f74e4032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f74f00096a0 tx=0x7f74f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:52.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.377+0000 7f74e37fe700 1 -- 172.21.15.23:0/4001332821 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74ec06b2b0 con 0x7f74fc0f4590 2021-11-19T08:36:52.526 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.541+0000 7f75016b0700 1 -- 172.21.15.23:0/4001332821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f74fc107f70 con 0x7f74fc0f4590 2021-11-19T08:36:52.673 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.687+0000 7f74e37fe700 1 -- 172.21.15.23:0/4001332821 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f74ec0126f0 con 0x7f74fc0f4590 2021-11-19T08:36:52.675 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.689+0000 7f75016b0700 1 -- 172.21.15.23:0/4001332821 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f74e402ff60 msgr2=0x7f74e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:52.675 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.689+0000 7f75016b0700 1 --2- 172.21.15.23:0/4001332821 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f74e402ff60 0x7f74e4032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f74f00096a0 tx=0x7f74f0006b40).stop 2021-11-19T08:36:52.675 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.689+0000 7f75016b0700 1 -- 172.21.15.23:0/4001332821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74fc0f4590 msgr2=0x7f74fc106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:52.676 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.689+0000 7f75016b0700 1 --2- 172.21.15.23:0/4001332821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74fc0f4590 0x7f74fc106580 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f74ec013510 tx=0x7f74ec00b040).stop 2021-11-19T08:36:52.676 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.689+0000 7f75016b0700 1 -- 172.21.15.23:0/4001332821 shutdown_connections 2021-11-19T08:36:52.676 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.689+0000 7f75016b0700 1 --2- 172.21.15.23:0/4001332821 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f74e402ff60 0x7f74e4032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:52.676 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.690+0000 7f75016b0700 1 --2- 172.21.15.23:0/4001332821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74fc0f4590 0x7f74fc106580 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:52.677 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.690+0000 7f75016b0700 1 -- 172.21.15.23:0/4001332821 >> 172.21.15.23:0/4001332821 conn(0x7f74fc0f00b0 msgr2=0x7f74fc0f0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:52.677 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.690+0000 7f75016b0700 1 -- 172.21.15.23:0/4001332821 shutdown_connections 2021-11-19T08:36:52.678 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:36:52.690+0000 7f75016b0700 1 -- 172.21.15.23:0/4001332821 wait complete. 2021-11-19T08:36:52.678 INFO:teuthology.orchestra.run.smithi023.stderr:adjusted tunables profile to default 2021-11-19T08:36:53.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:52 smithi023 conmon[33490]: cluster 2021-11-19T08:36:50.789026+0000 mgr.smithi023.ehlpcl (mgr.14164) 19 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:36:53.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:52 smithi023 conmon[33490]: cephadm 2021-11-19T08:36:51.557977+0000 mgr.smithi023.ehlpcl (mgr.14164) 20 : cephadm [INF] Deploying daemon prometheus.smithi023 on smithi023 2021-11-19T08:36:53.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:52 smithi023 conmon[33490]: audit 2021-11-19T08:36:52.542445+0000 mon.smithi023 (mon.0) 130 : audit [INF] from='client.? 172.21.15.23:0/4001332821' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2021-11-19T08:36:53.212 INFO:tasks.cephadm:Adding mon.smithi023 on smithi023 2021-11-19T08:36:53.213 INFO:tasks.cephadm:Adding mon.smithi134 on smithi134 2021-11-19T08:36:53.213 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph orch apply mon '2;smithi023:172.21.15.23=smithi023;smithi134:172.21.15.134=smithi134' 2021-11-19T08:36:54.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:53 smithi023 conmon[33490]: audit 2021-11-19T08:36:52.688612+0000 mon.smithi023 ( 2021-11-19T08:36:54.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:53 smithi023 conmon[33490]: mon.0) 131 : audit [INF] from='client.? 172.21.15.23:0/4001332821' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2021-11-19T08:36:54.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:53 smithi023 conmon[33490]: cluster 2021-11-19T08:36:52.688664+0000 mon.smithi023 (mon.0) 132 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2021-11-19T08:36:54.579 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.588+0000 7f4b971f5700 1 -- 172.21.15.134:0/2454696891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b90060280 msgr2=0x7f4b900f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:54.579 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.588+0000 7f4b971f5700 1 --2- 172.21.15.134:0/2454696891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b90060280 0x7f4b900f90f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f4b80009960 tx=0x7f4b80004d10).stop 2021-11-19T08:36:54.579 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.588+0000 7f4b971f5700 1 -- 172.21.15.134:0/2454696891 shutdown_connections 2021-11-19T08:36:54.580 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.588+0000 7f4b971f5700 1 --2- 172.21.15.134:0/2454696891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b90060280 0x7f4b900f90f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:54.580 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.588+0000 7f4b971f5700 1 -- 172.21.15.134:0/2454696891 >> 172.21.15.134:0/2454696891 conn(0x7f4b900f0b70 msgr2=0x7f4b900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:54.580 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.588+0000 7f4b971f5700 1 -- 172.21.15.134:0/2454696891 shutdown_connections 2021-11-19T08:36:54.580 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.588+0000 7f4b971f5700 1 -- 172.21.15.134:0/2454696891 wait complete. 2021-11-19T08:36:54.581 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.589+0000 7f4b971f5700 1 Processor -- start 2021-11-19T08:36:54.581 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.589+0000 7f4b971f5700 1 -- start start 2021-11-19T08:36:54.581 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.589+0000 7f4b971f5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b90060280 0x7f4b900f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:54.581 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.589+0000 7f4b971f5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b900f4ff0 con 0x7f4b90060280 2021-11-19T08:36:54.582 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.589+0000 7f4b94f91700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b90060280 0x7f4b900f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:54.582 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.589+0000 7f4b94f91700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b90060280 0x7f4b900f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.134:51324/0 (socket says 172.21.15.134:51324) 2021-11-19T08:36:54.582 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.589+0000 7f4b94f91700 1 -- 172.21.15.134:0/1415509854 learned_addr learned my addr 172.21.15.134:0/1415509854 (peer_addr_for_me v2:172.21.15.134:0/0) 2021-11-19T08:36:54.582 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.589+0000 7f4b94f91700 1 -- 172.21.15.134:0/1415509854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b80009640 con 0x7f4b90060280 2021-11-19T08:36:54.583 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.590+0000 7f4b94f91700 1 --2- 172.21.15.134:0/1415509854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b90060280 0x7f4b900f6b70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f4b80007b20 tx=0x7f4b80004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:54.583 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.590+0000 7f4b8d7fa700 1 -- 172.21.15.134:0/1415509854 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b80018070 con 0x7f4b90060280 2021-11-19T08:36:54.583 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.590+0000 7f4b971f5700 1 -- 172.21.15.134:0/1415509854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b900f5190 con 0x7f4b90060280 2021-11-19T08:36:54.583 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.590+0000 7f4b8d7fa700 1 -- 172.21.15.134:0/1415509854 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b80007e60 con 0x7f4b90060280 2021-11-19T08:36:54.583 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.590+0000 7f4b8d7fa700 1 -- 172.21.15.134:0/1415509854 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b8001d420 con 0x7f4b90060280 2021-11-19T08:36:54.584 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.590+0000 7f4b971f5700 1 -- 172.21.15.134:0/1415509854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b900f5620 con 0x7f4b90060280 2021-11-19T08:36:54.584 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.590+0000 7f4b8d7fa700 1 -- 172.21.15.134:0/1415509854 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b8001d580 con 0x7f4b90060280 2021-11-19T08:36:54.584 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.591+0000 7f4b8d7fa700 1 --2- 172.21.15.134:0/1415509854 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4b7c030000 0x7f4b7c0324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:54.584 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.591+0000 7f4b8d7fa700 1 -- 172.21.15.134:0/1415509854 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b80041730 con 0x7f4b90060280 2021-11-19T08:36:54.585 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.591+0000 7f4b971f5700 1 -- 172.21.15.134:0/1415509854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b90059100 con 0x7f4b90060280 2021-11-19T08:36:54.585 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.591+0000 7f4b8ffff700 1 --2- 172.21.15.134:0/1415509854 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4b7c030000 0x7f4b7c0324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:54.585 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.591+0000 7f4b8ffff700 1 --2- 172.21.15.134:0/1415509854 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4b7c030000 0x7f4b7c0324c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4b780096a0 tx=0x7f4b78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:54.586 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.596+0000 7f4b8d7fa700 1 -- 172.21.15.134:0/1415509854 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b8001d980 con 0x7f4b90060280 2021-11-19T08:36:54.755 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.770+0000 7f4b971f5700 1 -- 172.21.15.134:0/1415509854 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi023:172.21.15.23=smithi023;smithi134:172.21.15.134=smithi134", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b900f5e40 con 0x7f4b7c030000 2021-11-19T08:36:54.762 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.777+0000 7f4b8d7fa700 1 -- 172.21.15.134:0/1415509854 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4b900f5e40 con 0x7f4b7c030000 2021-11-19T08:36:54.763 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled mon update... 2021-11-19T08:36:54.764 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.779+0000 7f4b971f5700 1 -- 172.21.15.134:0/1415509854 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4b7c030000 msgr2=0x7f4b7c0324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:54.765 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.779+0000 7f4b971f5700 1 --2- 172.21.15.134:0/1415509854 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4b7c030000 0x7f4b7c0324c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4b780096a0 tx=0x7f4b78006b40).stop 2021-11-19T08:36:54.765 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.779+0000 7f4b971f5700 1 -- 172.21.15.134:0/1415509854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b90060280 msgr2=0x7f4b900f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:54.766 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.779+0000 7f4b971f5700 1 --2- 172.21.15.134:0/1415509854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b90060280 0x7f4b900f6b70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f4b80007b20 tx=0x7f4b80004d10).stop 2021-11-19T08:36:54.766 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.779+0000 7f4b971f5700 1 -- 172.21.15.134:0/1415509854 shutdown_connections 2021-11-19T08:36:54.766 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.779+0000 7f4b971f5700 1 --2- 172.21.15.134:0/1415509854 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4b7c030000 0x7f4b7c0324c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:54.767 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.779+0000 7f4b971f5700 1 --2- 172.21.15.134:0/1415509854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b90060280 0x7f4b900f6b70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:54.767 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.779+0000 7f4b971f5700 1 -- 172.21.15.134:0/1415509854 >> 172.21.15.134:0/1415509854 conn(0x7f4b900f0b70 msgr2=0x7f4b900f1680 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:54.768 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.779+0000 7f4b971f5700 1 -- 172.21.15.134:0/1415509854 shutdown_connections 2021-11-19T08:36:54.768 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:54.779+0000 7f4b971f5700 1 -- 172.21.15.134:0/1415509854 wait complete. 2021-11-19T08:36:55.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:54 smithi023 conmon[33490]: cluster 2021-11-19T08:36:52. 2021-11-19T08:36:55.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:54 smithi023 conmon[33490]: 789294+0000 mgr.smithi023.ehlpcl (mgr.14164) 21 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:36:55.151 DEBUG:teuthology.orchestra.run.smithi134:mon.smithi134> sudo journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi134.service 2021-11-19T08:36:55.154 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-19T08:36:55.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-19T08:36:55.181 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:-- Logs begin at Fri 2021-11-19 08:24:18 UTC. -- 2021-11-19T08:36:56.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:55 smithi023 conmon[33490]: audit 2021-11-19T08:36:54.771704+0000 mgr.smithi023.ehlpcl (mgr.14164) 22 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi023:172.21.15.23=smithi023;smithi134:172.21.15.134=smithi134", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:36:56.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:55 smithi023 conmon[33490]: audit 2021-11-19T08:36:54.777653+0000 mon.smithi023 (mon.0) 133 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:36:56.610 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.620+0000 7f6456f0c700 1 -- 172.21.15.134:0/1581897049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64500f51a0 msgr2=0x7f64500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:56.611 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.620+0000 7f6456f0c700 1 --2- 172.21.15.134:0/1581897049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64500f51a0 0x7f64500f55c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f644c009fb0 tx=0x7f644c009b30).stop 2021-11-19T08:36:56.611 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.620+0000 7f6456f0c700 1 -- 172.21.15.134:0/1581897049 shutdown_connections 2021-11-19T08:36:56.611 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.620+0000 7f6456f0c700 1 --2- 172.21.15.134:0/1581897049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64500f51a0 0x7f64500f55c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:56.612 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.620+0000 7f6456f0c700 1 -- 172.21.15.134:0/1581897049 >> 172.21.15.134:0/1581897049 conn(0x7f64500f09a0 msgr2=0x7f64500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:56.612 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.620+0000 7f6456f0c700 1 -- 172.21.15.134:0/1581897049 shutdown_connections 2021-11-19T08:36:56.613 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.620+0000 7f6456f0c700 1 -- 172.21.15.134:0/1581897049 wait complete. 2021-11-19T08:36:56.613 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.621+0000 7f6456f0c700 1 Processor -- start 2021-11-19T08:36:56.614 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.621+0000 7f6456f0c700 1 -- start start 2021-11-19T08:36:56.614 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.621+0000 7f6456f0c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64500f51a0 0x7f64501091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:56.614 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.621+0000 7f6456f0c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6450060240 con 0x7f64500f51a0 2021-11-19T08:36:56.615 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.621+0000 7f6455f0a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64500f51a0 0x7f64501091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:56.615 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.621+0000 7f6455f0a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64500f51a0 0x7f64501091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.134:51330/0 (socket says 172.21.15.134:51330) 2021-11-19T08:36:56.616 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.621+0000 7f6455f0a700 1 -- 172.21.15.134:0/3363277385 learned_addr learned my addr 172.21.15.134:0/3363277385 (peer_addr_for_me v2:172.21.15.134:0/0) 2021-11-19T08:36:56.616 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.622+0000 7f6455f0a700 1 -- 172.21.15.134:0/3363277385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c005040 con 0x7f64500f51a0 2021-11-19T08:36:56.617 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.622+0000 7f6455f0a700 1 --2- 172.21.15.134:0/3363277385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64500f51a0 0x7f64501091b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f644c009830 tx=0x7f644c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:56.617 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.622+0000 7f64467fc700 1 -- 172.21.15.134:0/3363277385 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f644c009320 con 0x7f64500f51a0 2021-11-19T08:36:56.618 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.622+0000 7f64467fc700 1 -- 172.21.15.134:0/3363277385 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f644c0136c0 con 0x7f64500f51a0 2021-11-19T08:36:56.619 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.622+0000 7f6456f0c700 1 -- 172.21.15.134:0/3363277385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6450060440 con 0x7f64500f51a0 2021-11-19T08:36:56.620 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.622+0000 7f64467fc700 1 -- 172.21.15.134:0/3363277385 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f644c01ee90 con 0x7f64500f51a0 2021-11-19T08:36:56.620 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.622+0000 7f6456f0c700 1 -- 172.21.15.134:0/3363277385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6450109860 con 0x7f64500f51a0 2021-11-19T08:36:56.620 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.623+0000 7f64467fc700 1 -- 172.21.15.134:0/3363277385 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f644c01e420 con 0x7f64500f51a0 2021-11-19T08:36:56.621 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.623+0000 7f6456f0c700 1 -- 172.21.15.134:0/3363277385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6450045ad0 con 0x7f64500f51a0 2021-11-19T08:36:56.621 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.623+0000 7f64467fc700 1 --2- 172.21.15.134:0/3363277385 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f643c02ff10 0x7f643c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:56.621 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.623+0000 7f64467fc700 1 -- 172.21.15.134:0/3363277385 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f644c040f40 con 0x7f64500f51a0 2021-11-19T08:36:56.621 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.623+0000 7f6455709700 1 --2- 172.21.15.134:0/3363277385 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f643c02ff10 0x7f643c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:56.621 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.623+0000 7f6455709700 1 --2- 172.21.15.134:0/3363277385 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f643c02ff10 0x7f643c0323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f64400096a0 tx=0x7f6440006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:56.622 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.627+0000 7f64467fc700 1 -- 172.21.15.134:0/3363277385 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f644c06d2b0 con 0x7f64500f51a0 2021-11-19T08:36:56.817 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.831+0000 7f6456f0c700 1 -- 172.21.15.134:0/3363277385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f645010a0e0 con 0x7f64500f51a0 2021-11-19T08:36:56.818 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.832+0000 7f64467fc700 1 -- 172.21.15.134:0/3363277385 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f644c02c2d0 con 0x7f64500f51a0 2021-11-19T08:36:56.819 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:36:56.819 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"908b41ee-4913-11ec-8c2c-001a4aab830c","modified":"2021-11-19T08:35:12.680933Z","created":"2021-11-19T08:35:12.680933Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-19T08:36:56.820 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.833+0000 7f6456f0c700 1 -- 172.21.15.134:0/3363277385 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f643c02ff10 msgr2=0x7f643c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:56.820 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.833+0000 7f6456f0c700 1 --2- 172.21.15.134:0/3363277385 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f643c02ff10 0x7f643c0323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f64400096a0 tx=0x7f6440006b40).stop 2021-11-19T08:36:56.821 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.833+0000 7f6456f0c700 1 -- 172.21.15.134:0/3363277385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64500f51a0 msgr2=0x7f64501091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:56.821 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.833+0000 7f6456f0c700 1 --2- 172.21.15.134:0/3363277385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64500f51a0 0x7f64501091b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f644c009830 tx=0x7f644c00b040).stop 2021-11-19T08:36:56.821 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.833+0000 7f6456f0c700 1 -- 172.21.15.134:0/3363277385 shutdown_connections 2021-11-19T08:36:56.822 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.833+0000 7f6456f0c700 1 --2- 172.21.15.134:0/3363277385 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f643c02ff10 0x7f643c0323d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:56.822 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.833+0000 7f6456f0c700 1 --2- 172.21.15.134:0/3363277385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64500f51a0 0x7f64501091b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:56.822 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.833+0000 7f6456f0c700 1 -- 172.21.15.134:0/3363277385 >> 172.21.15.134:0/3363277385 conn(0x7f64500f09a0 msgr2=0x7f64500f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:56.822 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.833+0000 7f6456f0c700 1 -- 172.21.15.134:0/3363277385 shutdown_connections 2021-11-19T08:36:56.823 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:56.833+0000 7f6456f0c700 1 -- 172.21.15.134:0/3363277385 wait complete. 2021-11-19T08:36:56.823 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2021-11-19T08:36:56.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:56 smithi023 conmon[33490]: cephadm 2021-11-19T08:36:54.773066+0000 2021-11-19T08:36:56.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:56 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 23 : cephadm [INF] Saving service mon spec with placement smithi023:172.21.15.23=smithi023;smithi134:172.21.15.134=smithi134;count:2 2021-11-19T08:36:56.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:56 smithi023 conmon[33490]: cluster 2021-11-19T08:36:54.789666+0000 mgr.smithi023.ehlpcl (mgr.14164) 24 : cluster 2021-11-19T08:36:56.898 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:56 smithi023 conmon[33490]: [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:36:57.972 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:57 smithi023 conmon[33490]: audit 2021-11-19T08:36:56.832512+0000 mon.smithi023 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.134:0/3363277385' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-19T08:36:58.221 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-19T08:36:58.222 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-19T08:36:59.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:58 smithi023 conmon[33490]: cluster 2021-11-19T08:36:56.790001+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:36:59.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:58 smithi023 conmon[33490]: ) 25 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:36:59.636 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.649+0000 7f93a617a700 1 -- 172.21.15.134:0/1886257182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f51a0 msgr2=0x7f93a00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.649+0000 7f93a617a700 1 --2- 172.21.15.134:0/1886257182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f51a0 0x7f93a00f55c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f939c009fb0 tx=0x7f939c009b30).stop 2021-11-19T08:36:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.649+0000 7f93a617a700 1 -- 172.21.15.134:0/1886257182 shutdown_connections 2021-11-19T08:36:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.649+0000 7f93a617a700 1 --2- 172.21.15.134:0/1886257182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f51a0 0x7f93a00f55c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:59.637 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.649+0000 7f93a617a700 1 -- 172.21.15.134:0/1886257182 >> 172.21.15.134:0/1886257182 conn(0x7f93a00f09a0 msgr2=0x7f93a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.649+0000 7f93a617a700 1 -- 172.21.15.134:0/1886257182 shutdown_connections 2021-11-19T08:36:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.649+0000 7f93a617a700 1 -- 172.21.15.134:0/1886257182 wait complete. 2021-11-19T08:36:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.650+0000 7f93a617a700 1 Processor -- start 2021-11-19T08:36:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.650+0000 7f93a617a700 1 -- start start 2021-11-19T08:36:59.638 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.650+0000 7f93a617a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f51a0 0x7f93a0109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.650+0000 7f93a617a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0060280 con 0x7f93a00f51a0 2021-11-19T08:36:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.650+0000 7f93a5178700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f51a0 0x7f93a0109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.650+0000 7f93a5178700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f51a0 0x7f93a0109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.134:51336/0 (socket says 172.21.15.134:51336) 2021-11-19T08:36:59.639 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.650+0000 7f93a5178700 1 -- 172.21.15.134:0/4278217163 learned_addr learned my addr 172.21.15.134:0/4278217163 (peer_addr_for_me v2:172.21.15.134:0/0) 2021-11-19T08:36:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.651+0000 7f93a5178700 1 -- 172.21.15.134:0/4278217163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939c005040 con 0x7f93a00f51a0 2021-11-19T08:36:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.651+0000 7f93a5178700 1 --2- 172.21.15.134:0/4278217163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f51a0 0x7f93a0109170 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f939c013510 tx=0x7f939c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.651+0000 7f9395ffb700 1 -- 172.21.15.134:0/4278217163 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f939c009320 con 0x7f93a00f51a0 2021-11-19T08:36:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.651+0000 7f93a617a700 1 -- 172.21.15.134:0/4278217163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a0060480 con 0x7f93a00f51a0 2021-11-19T08:36:59.640 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.651+0000 7f9395ffb700 1 -- 172.21.15.134:0/4278217163 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f939c013f00 con 0x7f93a00f51a0 2021-11-19T08:36:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.651+0000 7f9395ffb700 1 -- 172.21.15.134:0/4278217163 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f939c014dc0 con 0x7f93a00f51a0 2021-11-19T08:36:59.641 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.651+0000 7f93a617a700 1 -- 172.21.15.134:0/4278217163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a01098a0 con 0x7f93a00f51a0 2021-11-19T08:36:59.643 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.652+0000 7f9395ffb700 1 -- 172.21.15.134:0/4278217163 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f939c014420 con 0x7f93a00f51a0 2021-11-19T08:36:59.643 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.652+0000 7f93a617a700 1 -- 172.21.15.134:0/4278217163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a0045ad0 con 0x7f93a00f51a0 2021-11-19T08:36:59.643 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.652+0000 7f9395ffb700 1 --2- 172.21.15.134:0/4278217163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f938c02ff60 0x7f938c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:36:59.643 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.652+0000 7f9395ffb700 1 -- 172.21.15.134:0/4278217163 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f939c041030 con 0x7f93a00f51a0 2021-11-19T08:36:59.644 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.652+0000 7f93a4977700 1 --2- 172.21.15.134:0/4278217163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f938c02ff60 0x7f938c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:36:59.644 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.653+0000 7f93a4977700 1 --2- 172.21.15.134:0/4278217163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f938c02ff60 0x7f938c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f93900096a0 tx=0x7f9390006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:36:59.644 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.657+0000 7f9395ffb700 1 -- 172.21.15.134:0/4278217163 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f939c01ccf0 con 0x7f93a00f51a0 2021-11-19T08:36:59.859 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.873+0000 7f93a617a700 1 -- 172.21.15.134:0/4278217163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93a00fabe0 con 0x7f93a00f51a0 2021-11-19T08:36:59.859 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.874+0000 7f9395ffb700 1 -- 172.21.15.134:0/4278217163 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f939c01cb10 con 0x7f93a00f51a0 2021-11-19T08:36:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:36:59.860 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"908b41ee-4913-11ec-8c2c-001a4aab830c","modified":"2021-11-19T08:35:12.680933Z","created":"2021-11-19T08:35:12.680933Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-19T08:36:59.861 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.875+0000 7f93a617a700 1 -- 172.21.15.134:0/4278217163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f938c02ff60 msgr2=0x7f938c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:59.861 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.875+0000 7f93a617a700 1 --2- 172.21.15.134:0/4278217163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f938c02ff60 0x7f938c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f93900096a0 tx=0x7f9390006b40).stop 2021-11-19T08:36:59.861 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.875+0000 7f93a617a700 1 -- 172.21.15.134:0/4278217163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f51a0 msgr2=0x7f93a0109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:36:59.863 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.875+0000 7f93a617a700 1 --2- 172.21.15.134:0/4278217163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f51a0 0x7f93a0109170 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f939c013510 tx=0x7f939c00b040).stop 2021-11-19T08:36:59.863 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.876+0000 7f93a617a700 1 -- 172.21.15.134:0/4278217163 shutdown_connections 2021-11-19T08:36:59.864 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.876+0000 7f93a617a700 1 --2- 172.21.15.134:0/4278217163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f938c02ff60 0x7f938c032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:59.864 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.876+0000 7f93a617a700 1 --2- 172.21.15.134:0/4278217163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f51a0 0x7f93a0109170 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:36:59.864 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.876+0000 7f93a617a700 1 -- 172.21.15.134:0/4278217163 >> 172.21.15.134:0/4278217163 conn(0x7f93a00f09a0 msgr2=0x7f93a00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:36:59.864 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.876+0000 7f93a617a700 1 -- 172.21.15.134:0/4278217163 shutdown_connections 2021-11-19T08:36:59.864 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:36:59.876+0000 7f93a617a700 1 -- 172.21.15.134:0/4278217163 wait complete. 2021-11-19T08:36:59.865 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2021-11-19T08:37:00.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:59 smithi023 conmon[33490]: cluster 2021-11-19T08: 2021-11-19T08:37:00.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:59 smithi023 conmon[33490]: 36:58.790200+0000 mgr.smithi023.ehlpcl (mgr.14164) 26 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:00.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:59 smithi023 conmon[33490]: audit 2021-11-19T08:36:58.798209+0000 mon.smithi023 (mon.0) 135 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:00.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:59 smithi023 conmon[33490]: audit 2021-11-19T08:36:58.798775+0000 mon.smithi023 (mon.0) 136 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:37:00.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:59 smithi023 conmon[33490]: audit 2021-11-19T08:36:59.317729+0000 mon.smithi023 (mon.0) 137 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:00.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:36:59 smithi023 conmon[33490]: audit 2021-11-19T08:36:59.760079+0000 mon.smithi023 (mon.0) 138 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:01.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:00 smithi023 conmon[33490]: audit 2021-11-19T08:36:59.874572+0000 mon.smithi023 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.134:0/4278217163' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-19T08:37:01.638 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-19T08:37:01.638 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-19T08:37:02.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:01 smithi023 conmon[33490]: cluster 2021-11-19T08:37:00.790575+0000 mgr.smithi023.ehlpcl (mgr.14164) 27 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:03.302 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.316+0000 7fcad61ca700 1 -- 172.21.15.134:0/948687544 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcad00f6950 msgr2=0x7fcad00f6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:03.302 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.316+0000 7fcad61ca700 1 --2- 172.21.15.134:0/948687544 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcad00f6950 0x7fcad00f6d70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fcac0009fb0 tx=0x7fcac0009b30).stop 2021-11-19T08:37:03.302 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.316+0000 7fcad61ca700 1 -- 172.21.15.134:0/948687544 shutdown_connections 2021-11-19T08:37:03.303 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.316+0000 7fcad61ca700 1 --2- 172.21.15.134:0/948687544 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcad00f6950 0x7fcad00f6d70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:03.303 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.316+0000 7fcad61ca700 1 -- 172.21.15.134:0/948687544 >> 172.21.15.134:0/948687544 conn(0x7fcad00f0260 msgr2=0x7fcad00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:37:03.303 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.316+0000 7fcad61ca700 1 -- 172.21.15.134:0/948687544 shutdown_connections 2021-11-19T08:37:03.303 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.316+0000 7fcad61ca700 1 -- 172.21.15.134:0/948687544 wait complete. 2021-11-19T08:37:03.306 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.317+0000 7fcad61ca700 1 Processor -- start 2021-11-19T08:37:03.306 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.317+0000 7fcad61ca700 1 -- start start 2021-11-19T08:37:03.306 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.317+0000 7fcad61ca700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcad00f6950 0x7fcad010cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:03.307 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.317+0000 7fcad61ca700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad010d270 con 0x7fcad00f6950 2021-11-19T08:37:03.307 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.317+0000 7fcacf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcad00f6950 0x7fcad010cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:03.307 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.317+0000 7fcacf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcad00f6950 0x7fcad010cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.134:51342/0 (socket says 172.21.15.134:51342) 2021-11-19T08:37:03.307 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.317+0000 7fcacf7fe700 1 -- 172.21.15.134:0/1124563269 learned_addr learned my addr 172.21.15.134:0/1124563269 (peer_addr_for_me v2:172.21.15.134:0/0) 2021-11-19T08:37:03.308 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.318+0000 7fcacf7fe700 1 -- 172.21.15.134:0/1124563269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac0005040 con 0x7fcad00f6950 2021-11-19T08:37:03.308 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.318+0000 7fcacf7fe700 1 --2- 172.21.15.134:0/1124563269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcad00f6950 0x7fcad010cd80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fcac0013510 tx=0x7fcac000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:37:03.308 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.318+0000 7fcab7fff700 1 -- 172.21.15.134:0/1124563269 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcac0009320 con 0x7fcad00f6950 2021-11-19T08:37:03.308 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.318+0000 7fcad61ca700 1 -- 172.21.15.134:0/1124563269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad010d470 con 0x7fcad00f6950 2021-11-19T08:37:03.309 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.318+0000 7fcab7fff700 1 -- 172.21.15.134:0/1124563269 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcac0013f00 con 0x7fcad00f6950 2021-11-19T08:37:03.309 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.318+0000 7fcab7fff700 1 -- 172.21.15.134:0/1124563269 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcac0014d90 con 0x7fcad00f6950 2021-11-19T08:37:03.309 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.318+0000 7fcad61ca700 1 -- 172.21.15.134:0/1124563269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad010e0b0 con 0x7fcad00f6950 2021-11-19T08:37:03.309 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.319+0000 7fcab7fff700 1 -- 172.21.15.134:0/1124563269 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcac00263a0 con 0x7fcad00f6950 2021-11-19T08:37:03.310 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.319+0000 7fcab7fff700 1 --2- 172.21.15.134:0/1124563269 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcab802ff60 0x7fcab8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:03.310 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.319+0000 7fcab7fff700 1 -- 172.21.15.134:0/1124563269 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcac0040ff0 con 0x7fcad00f6950 2021-11-19T08:37:03.310 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.319+0000 7fcad61ca700 1 -- 172.21.15.134:0/1124563269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad010d900 con 0x7fcad00f6950 2021-11-19T08:37:03.310 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.319+0000 7fcaceffd700 1 --2- 172.21.15.134:0/1124563269 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcab802ff60 0x7fcab8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:03.311 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.320+0000 7fcaceffd700 1 --2- 172.21.15.134:0/1124563269 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcab802ff60 0x7fcab8032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fcac40096a0 tx=0x7fcac4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:37:03.311 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.324+0000 7fcab7fff700 1 -- 172.21.15.134:0/1124563269 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcac0012920 con 0x7fcad00f6950 2021-11-19T08:37:03.528 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.542+0000 7fcad61ca700 1 -- 172.21.15.134:0/1124563269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcad010e710 con 0x7fcad00f6950 2021-11-19T08:37:03.528 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.543+0000 7fcab7fff700 1 -- 172.21.15.134:0/1124563269 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcac002b020 con 0x7fcad00f6950 2021-11-19T08:37:03.529 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:37:03.529 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"908b41ee-4913-11ec-8c2c-001a4aab830c","modified":"2021-11-19T08:35:12.680933Z","created":"2021-11-19T08:35:12.680933Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-19T08:37:03.530 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.545+0000 7fcad61ca700 1 -- 172.21.15.134:0/1124563269 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcab802ff60 msgr2=0x7fcab8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:03.531 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.545+0000 7fcad61ca700 1 --2- 172.21.15.134:0/1124563269 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcab802ff60 0x7fcab8032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fcac40096a0 tx=0x7fcac4006b40).stop 2021-11-19T08:37:03.531 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.545+0000 7fcad61ca700 1 -- 172.21.15.134:0/1124563269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcad00f6950 msgr2=0x7fcad010cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:03.531 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.545+0000 7fcad61ca700 1 --2- 172.21.15.134:0/1124563269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcad00f6950 0x7fcad010cd80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fcac0013510 tx=0x7fcac000b040).stop 2021-11-19T08:37:03.531 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.545+0000 7fcad61ca700 1 -- 172.21.15.134:0/1124563269 shutdown_connections 2021-11-19T08:37:03.531 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.545+0000 7fcad61ca700 1 --2- 172.21.15.134:0/1124563269 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcab802ff60 0x7fcab8032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:03.532 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.545+0000 7fcad61ca700 1 --2- 172.21.15.134:0/1124563269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcad00f6950 0x7fcad010cd80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:03.532 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.545+0000 7fcad61ca700 1 -- 172.21.15.134:0/1124563269 >> 172.21.15.134:0/1124563269 conn(0x7fcad00f0260 msgr2=0x7fcad00f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:37:03.532 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.545+0000 7fcad61ca700 1 -- 172.21.15.134:0/1124563269 shutdown_connections 2021-11-19T08:37:03.532 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:03.545+0000 7fcad61ca700 1 -- 172.21.15.134:0/1124563269 wait complete. 2021-11-19T08:37:03.533 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2021-11-19T08:37:04.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:03 smithi023 conmon[33490]: cluster 2021-11-19T08:37:02.790934+0000 mgr.smithi023.ehlpcl (mgr.14164) 28 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:04.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:03 smithi023 conmon[33490]: audit 2021 2021-11-19T08:37:04.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:03 smithi023 conmon[33490]: -11-19T08:37:02.892407+0000 mon.smithi023 (mon.0) 140 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:04.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:03 smithi023 conmon[33490]: audit 2021-11-19T08:37:03.460266+0000 mon.smithi023 (mon.0) 141 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:04.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:03 smithi023 conmon[33490]: audit 2021-11- 2021-11-19T08:37:04.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:03 smithi023 conmon[33490]: 19T08:37:03.543818+0000 mon.smithi023 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.134:0/1124563269' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-19T08:37:04.879 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-19T08:37:04.880 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-19T08:37:05.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:05 smithi023 conmon[33490]: audit 2021-11-19T08:37:04.003327+0000 mon.smithi023 (mon.0) 143 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:05.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:05 smithi023 conmon[33490]: audit 2021-11-19T08:37:05.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:05 smithi023 conmon[33490]: 2021-11-19T08:37:04.006477+0000 mon.smithi023 (mon.0) 144 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:05.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:05 smithi023 conmon[33490]: audit 2021-11-19T08:37:04.007987+0000 mon.smithi023 (mon.0) 145 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-19T08:37:05.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:05 smithi023 conmon[33490]: audit 2021-11-19T08:37:04.010403+0000 mon.smithi023 (mon.0) 2021-11-19T08:37:05.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:05 smithi023 conmon[33490]: 146 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2021-11-19T08:37:05.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:05 smithi023 conmon[33490]: audit 2021-11-19T 2021-11-19T08:37:05.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:05 smithi023 conmon[33490]: 08:37:04.011294+0000 mon.smithi023 (mon. 2021-11-19T08:37:05.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:05 smithi023 conmon[33490]: 0) 147 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:05.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:05 smithi023 conmon[33490]: cephadm 2021-11 2021-11-19T08:37:05.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:05 smithi023 conmon[33490]: -19T08:37:04.012312+0000 2021-11-19T08:37:05.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:05 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 29 : cephadm [INF] 2021-11-19T08:37:05.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:05 smithi023 conmon[33490]: Deploying daemon crash.smithi134 on smithi134 2021-11-19T08:37:06.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:06 smithi023 conmon[33490]: cluster 2021-11-19T08:37:04.791306+0000 2021-11-19T08:37:06.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:06 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 30 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:06.684 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.693+0000 7f61d7898700 1 -- 172.21.15.134:0/351672305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f6510 msgr2=0x7f61d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:06.684 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.693+0000 7f61d7898700 1 --2- 172.21.15.134:0/351672305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f6510 0x7f61d00f6930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f61c0009fb0 tx=0x7f61c0009b30).stop 2021-11-19T08:37:06.685 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.693+0000 7f61d7898700 1 -- 172.21.15.134:0/351672305 shutdown_connections 2021-11-19T08:37:06.685 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.693+0000 7f61d7898700 1 --2- 172.21.15.134:0/351672305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f6510 0x7f61d00f6930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:06.685 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.693+0000 7f61d7898700 1 -- 172.21.15.134:0/351672305 >> 172.21.15.134:0/351672305 conn(0x7f61d00f1c70 msgr2=0x7f61d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:37:06.686 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.693+0000 7f61d7898700 1 -- 172.21.15.134:0/351672305 shutdown_connections 2021-11-19T08:37:06.686 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.693+0000 7f61d7898700 1 -- 172.21.15.134:0/351672305 wait complete. 2021-11-19T08:37:06.686 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.694+0000 7f61d7898700 1 Processor -- start 2021-11-19T08:37:06.687 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.694+0000 7f61d7898700 1 -- start start 2021-11-19T08:37:06.687 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.694+0000 7f61d7898700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f6510 0x7f61d006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:06.687 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.694+0000 7f61d7898700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d0069e20 con 0x7f61d00f6510 2021-11-19T08:37:06.688 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.695+0000 7f61d5634700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f6510 0x7f61d006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:06.688 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.695+0000 7f61d5634700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f6510 0x7f61d006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.134:51348/0 (socket says 172.21.15.134:51348) 2021-11-19T08:37:06.688 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.695+0000 7f61d5634700 1 -- 172.21.15.134:0/3802209506 learned_addr learned my addr 172.21.15.134:0/3802209506 (peer_addr_for_me v2:172.21.15.134:0/0) 2021-11-19T08:37:06.689 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.695+0000 7f61d5634700 1 -- 172.21.15.134:0/3802209506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c0005040 con 0x7f61d00f6510 2021-11-19T08:37:06.689 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.695+0000 7f61d5634700 1 --2- 172.21.15.134:0/3802209506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f6510 0x7f61d006b9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f61c0013510 tx=0x7f61c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:37:06.690 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.695+0000 7f61c5ffb700 1 -- 172.21.15.134:0/3802209506 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61c0009320 con 0x7f61d00f6510 2021-11-19T08:37:06.690 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.695+0000 7f61d7898700 1 -- 172.21.15.134:0/3802209506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d0069fc0 con 0x7f61d00f6510 2021-11-19T08:37:06.690 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.696+0000 7f61d7898700 1 -- 172.21.15.134:0/3802209506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d006a450 con 0x7f61d00f6510 2021-11-19T08:37:06.691 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.696+0000 7f61c5ffb700 1 -- 172.21.15.134:0/3802209506 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61c0013f00 con 0x7f61d00f6510 2021-11-19T08:37:06.691 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.696+0000 7f61c5ffb700 1 -- 172.21.15.134:0/3802209506 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61c0014d60 con 0x7f61d00f6510 2021-11-19T08:37:06.691 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.696+0000 7f61d7898700 1 -- 172.21.15.134:0/3802209506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b0004fa0 con 0x7f61d00f6510 2021-11-19T08:37:06.692 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.697+0000 7f61c5ffb700 1 -- 172.21.15.134:0/3802209506 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61c00263a0 con 0x7f61d00f6510 2021-11-19T08:37:06.692 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.697+0000 7f61c5ffb700 1 --2- 172.21.15.134:0/3802209506 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61bc02ffb0 0x7f61bc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:06.692 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.697+0000 7f61c5ffb700 1 -- 172.21.15.134:0/3802209506 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61c0040f60 con 0x7f61d00f6510 2021-11-19T08:37:06.692 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.697+0000 7f61d4e33700 1 --2- 172.21.15.134:0/3802209506 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61bc02ffb0 0x7f61bc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:06.693 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.697+0000 7f61d4e33700 1 --2- 172.21.15.134:0/3802209506 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61bc02ffb0 0x7f61bc032470 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f61cc0096a0 tx=0x7f61cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:37:06.693 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.701+0000 7f61c5ffb700 1 -- 172.21.15.134:0/3802209506 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61c00126f0 con 0x7f61d00f6510 2021-11-19T08:37:06.900 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.914+0000 7f61d7898700 1 -- 172.21.15.134:0/3802209506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61b0005670 con 0x7f61d00f6510 2021-11-19T08:37:06.901 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.915+0000 7f61c5ffb700 1 -- 172.21.15.134:0/3802209506 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f61c00126f0 con 0x7f61d00f6510 2021-11-19T08:37:06.902 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:37:06.903 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"908b41ee-4913-11ec-8c2c-001a4aab830c","modified":"2021-11-19T08:35:12.680933Z","created":"2021-11-19T08:35:12.680933Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-19T08:37:06.904 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.917+0000 7f61d7898700 1 -- 172.21.15.134:0/3802209506 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61bc02ffb0 msgr2=0x7f61bc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:06.904 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.917+0000 7f61d7898700 1 --2- 172.21.15.134:0/3802209506 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61bc02ffb0 0x7f61bc032470 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f61cc0096a0 tx=0x7f61cc006b40).stop 2021-11-19T08:37:06.905 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.917+0000 7f61d7898700 1 -- 172.21.15.134:0/3802209506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f6510 msgr2=0x7f61d006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:06.905 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.917+0000 7f61d7898700 1 --2- 172.21.15.134:0/3802209506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f6510 0x7f61d006b9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f61c0013510 tx=0x7f61c000b040).stop 2021-11-19T08:37:06.906 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.917+0000 7f61d7898700 1 -- 172.21.15.134:0/3802209506 shutdown_connections 2021-11-19T08:37:06.906 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.917+0000 7f61d7898700 1 --2- 172.21.15.134:0/3802209506 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61bc02ffb0 0x7f61bc032470 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:06.906 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.917+0000 7f61d7898700 1 --2- 172.21.15.134:0/3802209506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f6510 0x7f61d006b9a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:06.907 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.917+0000 7f61d7898700 1 -- 172.21.15.134:0/3802209506 >> 172.21.15.134:0/3802209506 conn(0x7f61d00f1c70 msgr2=0x7f61d00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:37:06.907 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.917+0000 7f61d7898700 1 -- 172.21.15.134:0/3802209506 shutdown_connections 2021-11-19T08:37:06.908 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:06.917+0000 7f61d7898700 1 -- 172.21.15.134:0/3802209506 wait complete. 2021-11-19T08:37:06.908 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2021-11-19T08:37:07.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: audit 2021-11-19T08:37:06.522671+0000 mon.smithi023 (mon.0) 148 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:07.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: audit 2021 2021-11-19T08:37:07.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: -11-19T08:37:06.524094+0000 mon.smithi023 (mon.0) 149 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.vitoqy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-19T08:37:07.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: audit 2021-11-19T08:37:06.526348+0000 mon.smithi023 (mon.0) 150 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.vitoqy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-11-19T08:37:07.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: audit 2021-11-19T08:37:06.527274+0000 mon.smithi023 (mon.0 2021-11-19T08:37:07.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: ) 151 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-19T08:37:07.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:37:07.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: T08:37:06.528299+0000 mon.smithi023 (mon 2021-11-19T08:37:07.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: .0) 152 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:07.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: cephadm 2021- 2021-11-19T08:37:07.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: 11-19T08:37:06.529251+0000 2021-11-19T08:37:07.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 31 : cephadm [INF] 2021-11-19T08:37:07.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: Deploying daemon mgr.smithi134.vitoqy on smithi134 2021-11-19T08:37:07.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: audit 2021-11-19T08:37: 2021-11-19T08:37:07.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: 06.915979+0000 mon.smithi023 (mon.0) 153 2021-11-19T08:37:07.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:07 smithi023 conmon[33490]: : audit [DBG] from='client.? 172.21.15.134:0/3802209506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-19T08:37:08.449 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-19T08:37:08.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-19T08:37:08.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:08 smithi023 conmon[33490]: cluster 2021-11-19T08:37:06.791729+0000 mgr.smithi023.ehlpcl (mgr.14164) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:10.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:09 smithi023 conmon[33490]: audit 2021-11-19T08:37:08.754839+0000 mon.smithi023 (mon.0) 154 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:10.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:09 smithi023 conmon[33490]: audit 2021-11-19T08:37:10.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:09 smithi023 conmon[33490]: 2021-11-19T08:37:08.756087+0000 mon.smithi023 (mon.0) 155 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-19T08:37:10.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:09 smithi023 conmon[33490]: audit 2021-11-19T08:37:08.757036+0000 mon.smithi023 (mon.0) 156 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:10.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:09 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:08.758125+0000 mgr.smithi023.ehlpcl (mgr.14164) 33 : cephadm [INF] Deploying daemon mon.smithi134 on smithi134 2021-11-19T08:37:10.376 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.388+0000 7fbd0dd57700 1 -- 172.21.15.134:0/1836019036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd080f62a0 msgr2=0x7fbd080f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:10.376 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.388+0000 7fbd0dd57700 1 --2- 172.21.15.134:0/1836019036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd080f62a0 0x7fbd080f66c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fbcfc009fb0 tx=0x7fbcfc009b30).stop 2021-11-19T08:37:10.377 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.388+0000 7fbd0dd57700 1 -- 172.21.15.134:0/1836019036 shutdown_connections 2021-11-19T08:37:10.377 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.388+0000 7fbd0dd57700 1 --2- 172.21.15.134:0/1836019036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd080f62a0 0x7fbd080f66c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:10.377 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.388+0000 7fbd0dd57700 1 -- 172.21.15.134:0/1836019036 >> 172.21.15.134:0/1836019036 conn(0x7fbd080f1aa0 msgr2=0x7fbd080f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:37:10.377 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.388+0000 7fbd0dd57700 1 -- 172.21.15.134:0/1836019036 shutdown_connections 2021-11-19T08:37:10.378 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.388+0000 7fbd0dd57700 1 -- 172.21.15.134:0/1836019036 wait complete. 2021-11-19T08:37:10.378 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.389+0000 7fbd0dd57700 1 Processor -- start 2021-11-19T08:37:10.378 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.389+0000 7fbd0dd57700 1 -- start start 2021-11-19T08:37:10.378 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.389+0000 7fbd0dd57700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd080f62a0 0x7fbd081064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:10.379 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.389+0000 7fbd0dd57700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd081069a0 con 0x7fbd080f62a0 2021-11-19T08:37:10.379 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.390+0000 7fbd0cd55700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd080f62a0 0x7fbd081064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:10.379 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.390+0000 7fbd0cd55700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd080f62a0 0x7fbd081064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.134:51358/0 (socket says 172.21.15.134:51358) 2021-11-19T08:37:10.379 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.390+0000 7fbd0cd55700 1 -- 172.21.15.134:0/542209912 learned_addr learned my addr 172.21.15.134:0/542209912 (peer_addr_for_me v2:172.21.15.134:0/0) 2021-11-19T08:37:10.380 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.390+0000 7fbd0cd55700 1 -- 172.21.15.134:0/542209912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcfc005040 con 0x7fbd080f62a0 2021-11-19T08:37:10.380 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.390+0000 7fbd0cd55700 1 --2- 172.21.15.134:0/542209912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd080f62a0 0x7fbd081064b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fbcfc013510 tx=0x7fbcfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:37:10.380 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.390+0000 7fbd057fa700 1 -- 172.21.15.134:0/542209912 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcfc009320 con 0x7fbd080f62a0 2021-11-19T08:37:10.380 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.390+0000 7fbd0dd57700 1 -- 172.21.15.134:0/542209912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd08106ba0 con 0x7fbd080f62a0 2021-11-19T08:37:10.380 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.390+0000 7fbd057fa700 1 -- 172.21.15.134:0/542209912 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcfc013f00 con 0x7fbd080f62a0 2021-11-19T08:37:10.381 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.391+0000 7fbd057fa700 1 -- 172.21.15.134:0/542209912 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcfc014dc0 con 0x7fbd080f62a0 2021-11-19T08:37:10.381 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.391+0000 7fbd0dd57700 1 -- 172.21.15.134:0/542209912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd08106fb0 con 0x7fbd080f62a0 2021-11-19T08:37:10.382 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.391+0000 7fbd057fa700 1 -- 172.21.15.134:0/542209912 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbcfc01d3d0 con 0x7fbd080f62a0 2021-11-19T08:37:10.383 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.391+0000 7fbd057fa700 1 --2- 172.21.15.134:0/542209912 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbcf8034370 0x7fbcf8036830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:10.383 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.391+0000 7fbd057fa700 1 -- 172.21.15.134:0/542209912 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbcfc041120 con 0x7fbd080f62a0 2021-11-19T08:37:10.383 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.392+0000 7fbd07fff700 1 --2- 172.21.15.134:0/542209912 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbcf8034370 0x7fbcf8036830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:10.383 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.392+0000 7fbd0dd57700 1 -- 172.21.15.134:0/542209912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd08059100 con 0x7fbd080f62a0 2021-11-19T08:37:10.384 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.392+0000 7fbd07fff700 1 --2- 172.21.15.134:0/542209912 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbcf8034370 0x7fbcf8036830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fbcf40096a0 tx=0x7fbcf4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:37:10.384 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.397+0000 7fbd057fa700 1 -- 172.21.15.134:0/542209912 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcfc012920 con 0x7fbd080f62a0 2021-11-19T08:37:10.609 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.623+0000 7fbd0dd57700 1 -- 172.21.15.134:0/542209912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd08107d50 con 0x7fbd080f62a0 2021-11-19T08:37:10.610 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.624+0000 7fbd057fa700 1 -- 172.21.15.134:0/542209912 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbcfc02b020 con 0x7fbd080f62a0 2021-11-19T08:37:10.610 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:37:10.610 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"908b41ee-4913-11ec-8c2c-001a4aab830c","modified":"2021-11-19T08:35:12.680933Z","created":"2021-11-19T08:35:12.680933Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2021-11-19T08:37:10.612 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.626+0000 7fbd0dd57700 1 -- 172.21.15.134:0/542209912 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbcf8034370 msgr2=0x7fbcf8036830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:10.612 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.626+0000 7fbd0dd57700 1 --2- 172.21.15.134:0/542209912 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbcf8034370 0x7fbcf8036830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fbcf40096a0 tx=0x7fbcf4006b40).stop 2021-11-19T08:37:10.612 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.626+0000 7fbd0dd57700 1 -- 172.21.15.134:0/542209912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd080f62a0 msgr2=0x7fbd081064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:10.612 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.626+0000 7fbd0dd57700 1 --2- 172.21.15.134:0/542209912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd080f62a0 0x7fbd081064b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fbcfc013510 tx=0x7fbcfc00b040).stop 2021-11-19T08:37:10.613 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.626+0000 7fbd0dd57700 1 -- 172.21.15.134:0/542209912 shutdown_connections 2021-11-19T08:37:10.613 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.626+0000 7fbd0dd57700 1 --2- 172.21.15.134:0/542209912 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbcf8034370 0x7fbcf8036830 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:10.613 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.626+0000 7fbd0dd57700 1 --2- 172.21.15.134:0/542209912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd080f62a0 0x7fbd081064b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:10.613 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.626+0000 7fbd0dd57700 1 -- 172.21.15.134:0/542209912 >> 172.21.15.134:0/542209912 conn(0x7fbd080f1aa0 msgr2=0x7fbd080f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:37:10.613 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.626+0000 7fbd0dd57700 1 -- 172.21.15.134:0/542209912 shutdown_connections 2021-11-19T08:37:10.614 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:10.626+0000 7fbd0dd57700 1 -- 172.21.15.134:0/542209912 wait complete. 2021-11-19T08:37:10.614 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2021-11-19T08:37:11.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:10 smithi023 conmon[33490]: cluster 2021-11-19T08:37:08.792082+0000 mgr.smithi023.ehlpcl (mgr.14164) 34 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:11.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:10 smithi023 conmon[33490]: audit 2021-11-19T08:37:10.624650+0000 mon.smithi023 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.134:0/542209912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-19T08:37:13.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:12 smithi023 conmon[33490]: cluster 2021-11-19T08:37:10.792454+0000 mgr.smithi023.ehlpcl (mgr.14164) 35 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:14.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:13 smithi023 conmon[33490]: cluster 2021-11-19T08:37:13.697574+0000 mon.smithi023 (mon.0) 158 : cluster [DBG] Standby manager daemon smithi134.vitoqy started 2021-11-19T08:37:14.248 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:13 smithi023 conmon[33490]: audit 2021-11-19T08:37:13.699081+0000 mon.smithi023 (mon.0) 159 : audit [DBG] from='mgr.? 172.21.15.134:0/1572643438' entity='mgr.smithi134.vitoqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.vitoqy/crt"}]: dispatch 2021-11-19T08:37:14.248 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:13 smithi023 conmon[33490]: audit 2021-11-19T08:37:13.699919+0000 mon.smithi023 ( 2021-11-19T08:37:14.249 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:13 smithi023 conmon[33490]: mon.0) 160 : audit [DBG] from='mgr.? 172.21.15.134:0/1572643438' entity='mgr.smithi134.vitoqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-11-19T08:37:14.249 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:13 smithi023 conmon[33490]: audit 2021-11-19T08:37:14.249 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:13 smithi023 conmon[33490]: 2021-11-19T08:37:13.702964+0000 mon.smithi023 (mon 2021-11-19T08:37:14.250 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:13 smithi023 conmon[33490]: .0) 161 : audit [DBG] from='mgr.? 172.21.15.134:0/1572643438' entity='mgr.smithi134.vitoqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.vitoqy/key"}]: dispatch 2021-11-19T08:37:14.250 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:13 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:37:14.251 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:13 smithi023 conmon[33490]: T08:37:13.703433+0000 mon.smithi023 ( 2021-11-19T08:37:14.251 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:13 smithi023 conmon[33490]: mon.0) 162 : audit [DBG] from='mgr.? 172.21.15.134:0/1572643438' entity='mgr.smithi134.vitoqy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-11-19T08:37:14.464 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-11-19T08:37:14.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph mon dump -f json 2021-11-19T08:37:15.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 systemd[1]: Starting Ceph mon.smithi134 for 908b41ee-4913-11ec-8c2c-001a4aab830c... 2021-11-19T08:37:15.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:14 smithi023 conmon[33490]: cluster 2021-11-19T08:37:12.792810+0000 mgr.smithi023.ehlpcl (mgr.14164) 36 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:15.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:14 smithi023 conmon[33490]: cluster 2021-11-19T08:37:15.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:14 smithi023 conmon[33490]: 2021-11-19T08:37:13.773372+0000 mon.smithi023 (mon.0) 163 : cluster [DBG] mgrmap e18: smithi023.ehlpcl(active, since 55s), standbys: smithi134.vitoqy 2021-11-19T08:37:15.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:14 smithi023 conmon[33490]: audit 2021-11-19T08:37:13.773547+0000 mon.smithi023 (mon.0) 164 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mgr metadata", "who": "smithi134.vitoqy", "id": "smithi134.vitoqy"}]: dispatch 2021-11-19T08:37:15.913 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 podman[36295]: 2021-11-19T08:37:15.914 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.832+0000 7f229b6fa700 0 set uid:gid to 167:167 (ceph:ceph) 2021-11-19T08:37:15.914 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.832+0000 7f229b6fa700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2021-11-19T08:37:15.914 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.832+0000 7f229b6fa700 0 pidfile_write: ignore empty --pid-file 2021-11-19T08:37:15.915 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 bash[36205]: 48f4b0e16c71b3f1fdfc6f61e7d017cdf40f69768b516948437e473ebaf7cb69 2021-11-19T08:37:15.915 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.836+0000 7f229b6fa700 0 load: jerasure load: lrc load: isa 2021-11-19T08:37:15.915 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: RocksDB version: 6.8.1 2021-11-19T08:37:15.915 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.916 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-11-19T08:37:15.916 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.916 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Compile date May 13 2021 2021-11-19T08:37:15.916 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: DB SUMMARY 2021-11-19T08:37:15.916 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.917 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: CURRENT file: CURRENT 2021-11-19T08:37:15.917 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.917 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: IDENTITY file: IDENTITY 2021-11-19T08:37:15.917 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.918 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2021-11-19T08:37:15.918 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.918 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi134/store.db dir, Total Num: 0, files: 2021-11-19T08:37:15.918 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.918 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi134/store.db: 000003.log size: 511 ; 2021-11-19T08:37:15.919 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.919 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.error_if_exists: 0 2021-11-19T08:37:15.920 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.920 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.create_if_missing: 0 2021-11-19T08:37:15.920 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.paranoid_checks: 1 2021-11-19T08:37:15.920 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.env: 0x56301126d1c0 2021-11-19T08:37:15.921 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.fs: Posix File System 2021-11-19T08:37:15.921 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.info_log: 0x56301298b960 2021-11-19T08:37:15.921 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.max_file_opening_threads: 16 2021-11-19T08:37:15.921 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.statistics: (nil) 2021-11-19T08:37:15.921 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.use_fsync: 0 2021-11-19T08:37:15.922 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.max_log_file_size: 0 2021-11-19T08:37:15.922 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-11-19T08:37:15.922 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-11-19T08:37:15.922 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.keep_log_file_num: 1000 2021-11-19T08:37:15.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.recycle_log_file_num: 0 2021-11-19T08:37:15.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.allow_fallocate: 1 2021-11-19T08:37:15.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.allow_mmap_reads: 0 2021-11-19T08:37:15.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.allow_mmap_writes: 0 2021-11-19T08:37:15.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.use_direct_reads: 0 2021-11-19T08:37:15.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-11-19T08:37:15.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.create_missing_column_families: 0 2021-11-19T08:37:15.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.db_log_dir: 2021-11-19T08:37:15.924 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi134/store.db 2021-11-19T08:37:15.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-11-19T08:37:15.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.max_subcompactions: 1 2021-11-19T08:37:15.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.max_background_flushes: -1 2021-11-19T08:37:15.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-11-19T08:37:15.925 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-11-19T08:37:15.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-11-19T08:37:15.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-11-19T08:37:15.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-11-19T08:37:15.926 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.advise_random_on_open: 1 2021-11-19T08:37:15.927 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.db_write_buffer_size: 0 2021-11-19T08:37:15.927 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.write_buffer_manager: 0x563013600030 2021-11-19T08:37:15.927 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-11-19T08:37:15.927 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-11-19T08:37:15.927 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-11-19T08:37:15.928 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-11-19T08:37:15.931 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.rate_limiter: (nil) 2021-11-19T08:37:15.931 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-11-19T08:37:15.931 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.wal_recovery_mode: 2 2021-11-19T08:37:15.932 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.enable_thread_tracking: 0 2021-11-19T08:37:15.932 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.enable_pipelined_write: 0 2021-11-19T08:37:15.932 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.unordered_write: 0 2021-11-19T08:37:15.932 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-11-19T08:37:15.932 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-11-19T08:37:15.933 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-11-19T08:37:15.933 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-11-19T08:37:15.933 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.row_cache: None 2021-11-19T08:37:15.933 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.wal_filter: None 2021-11-19T08:37:15.934 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-11-19T08:37:15.934 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.allow_ingest_behind: 0 2021-11-19T08:37:15.934 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.preserve_deletes: 0 2021-11-19T08:37:15.934 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.two_write_queues: 0 2021-11-19T08:37:15.934 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.manual_wal_flush: 0 2021-11-19T08:37:15.935 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.atomic_flush: 0 2021-11-19T08:37:15.935 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-11-19T08:37:15.935 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-11-19T08:37:15.935 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-11-19T08:37:15.936 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.log_readahead_size: 0 2021-11-19T08:37:15.936 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-11-19T08:37:15.936 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.max_background_jobs: 2 2021-11-19T08:37:15.936 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.max_background_compactions: -1 2021-11-19T08:37:15.936 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-11-19T08:37:15.937 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-11-19T08:37:15.937 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-11-19T08:37:15.937 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.max_total_wal_size: 0 2021-11-19T08:37:15.937 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-11-19T08:37:15.938 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-11-19T08:37:15.938 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-11-19T08:37:15.938 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-11-19T08:37:15.938 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.max_open_files: -1 2021-11-19T08:37:15.939 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.bytes_per_sync: 0 2021-11-19T08:37:15.939 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-11-19T08:37:15.939 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-11-19T08:37:15.939 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.compaction_readahead_size: 0 2021-11-19T08:37:15.939 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Compression algorithms supported: 2021-11-19T08:37:15.940 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-11-19T08:37:15.940 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: kZSTD supported: 0 2021-11-19T08:37:15.940 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: kXpressCompression supported: 0 2021-11-19T08:37:15.941 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: kLZ4HCCompression supported: 1 2021-11-19T08:37:15.941 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: kLZ4Compression supported: 1 2021-11-19T08:37:15.941 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: kBZip2Compression supported: 0 2021-11-19T08:37:15.941 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: kZlibCompression supported: 1 2021-11-19T08:37:15.942 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: kSnappyCompression supported: 1 2021-11-19T08:37:15.942 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-11-19T08:37:15.942 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi134/store.db/MANIFEST-000001 2021-11-19T08:37:15.942 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.943 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-11-19T08:37:15.943 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.943 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-11-19T08:37:15.943 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.merge_operator: 2021-11-19T08:37:15.944 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.compaction_filter: None 2021-11-19T08:37:15.944 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.compaction_filter_factory: None 2021-11-19T08:37:15.944 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.944 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-11-19T08:37:15.944 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-11-19T08:37:15.945 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5630128a2128) 2021-11-19T08:37:15.945 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: cache_index_and_filter_blocks: 1 2021-11-19T08:37:15.945 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: cache_index_and_filter_blocks_with_high_priority: 0 2021-11-19T08:37:15.945 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-11-19T08:37:15.946 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: pin_top_level_index_and_filter: 1 2021-11-19T08:37:15.946 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: index_type: 0 2021-11-19T08:37:15.946 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: data_block_index_type: 0 2021-11-19T08:37:15.946 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: index_shortening: 1 2021-11-19T08:37:15.946 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: data_block_hash_table_util_ratio: 0.750000 2021-11-19T08:37:15.947 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: hash_index_allow_collision: 1 2021-11-19T08:37:15.947 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: checksum: 1 2021-11-19T08:37:15.947 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: no_block_cache: 0 2021-11-19T08:37:15.947 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: block_cache: 0x5630128d8f10 2021-11-19T08:37:15.948 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: block_cache_name: BinnedLRUCache 2021-11-19T08:37:15.948 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: block_cache_options: 2021-11-19T08:37:15.948 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: capacity : 536870912 2021-11-19T08:37:15.948 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: num_shard_bits : 4 2021-11-19T08:37:15.948 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: strict_capacity_limit : 0 2021-11-19T08:37:15.949 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: high_pri_pool_ratio: 0.000 2021-11-19T08:37:15.949 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: block_cache_compressed: (nil) 2021-11-19T08:37:15.949 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: persistent_cache: (nil) 2021-11-19T08:37:15.949 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: block_size: 4096 2021-11-19T08:37:15.950 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: block_size_deviation: 10 2021-11-19T08:37:15.950 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: block_restart_interval: 16 2021-11-19T08:37:15.950 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: index_block_restart_interval: 1 2021-11-19T08:37:15.950 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: metadata_block_size: 4096 2021-11-19T08:37:15.950 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: partition_filters: 0 2021-11-19T08:37:15.951 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: use_delta_encoding: 1 2021-11-19T08:37:15.951 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: filter_policy: rocksdb.BuiltinBloomFilter 2021-11-19T08:37:15.951 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: whole_key_filtering: 1 2021-11-19T08:37:15.951 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: verify_compression: 0 2021-11-19T08:37:15.952 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: read_amp_bytes_per_bit: 0 2021-11-19T08:37:15.952 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: format_version: 2 2021-11-19T08:37:15.952 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: enable_index_compression: 1 2021-11-19T08:37:15.952 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: block_align: 0 2021-11-19T08:37:15.952 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.953 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.write_buffer_size: 33554432 2021-11-19T08:37:15.953 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.max_write_buffer_number: 2 2021-11-19T08:37:15.953 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.compression: NoCompression 2021-11-19T08:37:15.953 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.bottommost_compression: Disabled 2021-11-19T08:37:15.954 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.prefix_extractor: nullptr 2021-11-19T08:37:15.954 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.837+0000 7f229b6fa700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-11-19T08:37:15.954 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.num_levels: 7 2021-11-19T08:37:15.954 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-11-19T08:37:15.954 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-11-19T08:37:15.955 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-11-19T08:37:15.955 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-11-19T08:37:15.955 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-11-19T08:37:15.955 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-11-19T08:37:15.956 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-11-19T08:37:15.956 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-11-19T08:37:15.956 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-11-19T08:37:15.956 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-11-19T08:37:15.956 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compression_opts.level: 32767 2021-11-19T08:37:15.957 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compression_opts.strategy: 0 2021-11-19T08:37:15.957 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-11-19T08:37:15.957 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-11-19T08:37:15.957 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compression_opts.enabled: false 2021-11-19T08:37:15.958 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-11-19T08:37:15.958 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-11-19T08:37:15.958 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-11-19T08:37:15.958 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.target_file_size_base: 67108864 2021-11-19T08:37:15.958 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-11-19T08:37:15.959 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-11-19T08:37:15.959 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-11-19T08:37:15.959 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-11-19T08:37:15.959 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-11-19T08:37:15.960 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-11-19T08:37:15.960 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-11-19T08:37:15.960 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-11-19T08:37:15.960 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-11-19T08:37:15.960 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-11-19T08:37:15.961 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-11-19T08:37:15.961 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-11-19T08:37:15.961 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-11-19T08:37:15.961 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.arena_block_size: 4194304 2021-11-19T08:37:15.962 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-11-19T08:37:15.962 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-11-19T08:37:15.962 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-11-19T08:37:15.962 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.disable_auto_compactions: 0 2021-11-19T08:37:15.962 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-11-19T08:37:15.963 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-11-19T08:37:15.963 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-11-19T08:37:15.963 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-11-19T08:37:15.963 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-11-19T08:37:15.964 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-11-19T08:37:15.964 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-11-19T08:37:15.964 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-11-19T08:37:15.964 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-11-19T08:37:15.964 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-11-19T08:37:15.965 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.table_properties_collectors: 2021-11-19T08:37:15.965 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.inplace_update_support: 0 2021-11-19T08:37:15.965 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-11-19T08:37:15.965 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-11-19T08:37:15.966 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-11-19T08:37:15.966 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-11-19T08:37:15.966 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.bloom_locality: 0 2021-11-19T08:37:15.966 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.max_successive_merges: 0 2021-11-19T08:37:15.966 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-11-19T08:37:15.967 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.paranoid_file_checks: 0 2021-11-19T08:37:15.967 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.force_consistency_checks: 0 2021-11-19T08:37:15.967 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.report_bg_io_stats: 0 2021-11-19T08:37:15.967 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.ttl: 2592000 2021-11-19T08:37:15.968 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-11-19T08:37:15.968 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi134/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-11-19T08:37:15.968 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.968 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-11-19T08:37:15.968 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.969 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637311035839842, "job": 1, "event": "recovery_started", "log_files": [3]} 2021-11-19T08:37:15.969 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.969 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.838+0000 7f229b6fa700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2021-11-19T08:37:15.969 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.838+0000 7f229b6fa700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-11-19T08:37:15.970 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.839+0000 7f229b6fa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637311035840465, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1637311035, "oldest_key_time": 3, "file_creation_time": 0}} 2021-11-19T08:37:15.970 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.839+0000 7f229b6fa700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2021-11-19T08:37:15.970 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.970 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.839+0000 7f229b6fa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1637311035840965, "job": 1, "event": "recovery_finished"} 2021-11-19T08:37:15.970 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.840+0000 7f229b6fa700 4 rocksdb: DB pointer 0x56301299d800 2021-11-19T08:37:15.971 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.841+0000 7f2282512700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-11-19T08:37:15.971 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.841+0000 7f2282512700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-11-19T08:37:15.971 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: ** DB Stats ** 2021-11-19T08:37:15.971 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-19T08:37:15.972 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-11-19T08:37:15.972 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-11-19T08:37:15.972 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-19T08:37:15.972 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-11-19T08:37:15.972 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-11-19T08:37:15.973 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-11-19T08:37:15.973 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.973 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: ** Compaction Stats [default] ** 2021-11-19T08:37:15.973 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-19T08:37:15.974 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-19T08:37:15.974 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.4 0.00 0.00 1 0.001 0 0 2021-11-19T08:37:15.974 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.4 0.00 0.00 1 0.001 0 0 2021-11-19T08:37:15.974 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.4 0.00 0.00 1 0.001 0 0 2021-11-19T08:37:15.975 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.975 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: ** Compaction Stats [default] ** 2021-11-19T08:37:15.975 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-19T08:37:15.975 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-19T08:37:15.975 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.4 0.00 0.00 1 0.001 0 0 2021-11-19T08:37:15.976 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-19T08:37:15.976 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-19T08:37:15.976 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-19T08:37:15.976 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: AddFile(Total Files): cumulative 0, interval 0 2021-11-19T08:37:15.976 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-19T08:37:15.977 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: AddFile(Keys): cumulative 0, interval 0 2021-11-19T08:37:15.977 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Cumulative compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-19T08:37:15.977 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Interval compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-19T08:37:15.977 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-19T08:37:15.978 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.978 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: ** File Read Latency Histogram By Level [default] ** 2021-11-19T08:37:15.978 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.978 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: ** Compaction Stats [default] ** 2021-11-19T08:37:15.978 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-19T08:37:15.979 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-19T08:37:15.979 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.4 0.00 0.00 1 0.001 0 0 2021-11-19T08:37:15.979 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.4 0.00 0.00 1 0.001 0 0 2021-11-19T08:37:15.979 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-11-19T08:37:15.980 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.980 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: ** Compaction Stats [default] ** 2021-11-19T08:37:15.980 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-11-19T08:37:15.980 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-11-19T08:37:15.980 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.4 0.00 0.00 1 0.001 0 0 2021-11-19T08:37:15.981 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Uptime(secs): 0.0 total, 0.0 interval 2021-11-19T08:37:15.982 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Flush(GB): cumulative 0.000, interval 0.000 2021-11-19T08:37:15.982 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: AddFile(GB): cumulative 0.000, interval 0.000 2021-11-19T08:37:15.982 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: AddFile(Total Files): cumulative 0, interval 0 2021-11-19T08:37:15.982 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: AddFile(L0 Files): cumulative 0, interval 0 2021-11-19T08:37:15.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: AddFile(Keys): cumulative 0, interval 0 2021-11-19T08:37:15.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Cumulative compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-19T08:37:15.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-11-19T08:37:15.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-11-19T08:37:15.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: ** File Read Latency Histogram By Level [default] ** 2021-11-19T08:37:15.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.841+0000 7f229b6fa700 0 mon.smithi134 does not exist in monmap, will attempt to join an existing cluster 2021-11-19T08:37:15.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.841+0000 7f229b6fa700 0 using public_addr v2:172.21.15.134:0/0 -> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2021-11-19T08:37:15.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.841+0000 7f229b6fa700 0 starting mon.smithi134 rank -1 at public addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] at bind addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi134 fsid 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:37:15.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.843+0000 7f229b6fa700 1 mon.smithi134@-1(???) e0 preinit fsid 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:37:15.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.867+0000 7f2284d17700 0 mon.smithi134@-1(synchronizing).mds e1 new map 2021-11-19T08:37:15.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.867+0000 7f2284d17700 0 mon.smithi134@-1(synchronizing).mds e1 print_map 2021-11-19T08:37:15.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: e1 2021-11-19T08:37:15.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: enable_multiple, ever_enabled_multiple: 1,1 2021-11-19T08:37:15.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-19T08:37:15.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: legacy client fscid: -1 2021-11-19T08:37:15.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: No filesystems configured 2021-11-19T08:37:15.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.867+0000 7f2284d17700 1 mon.smithi134@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2021-11-19T08:37:15.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.867+0000 7f2284d17700 1 mon.smithi134@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2021-11-19T08:37:15.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.867+0000 7f2284d17700 1 mon.smithi134@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2021-11-19T08:37:15.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.867+0000 7f2284d17700 1 mon.smithi134@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2021-11-19T08:37:15.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.867+0000 7f2284d17700 1 mon.smithi134@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2021-11-19T08:37:15.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.867+0000 7f2284d17700 1 mon.smithi134@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2021-11-19T08:37:15.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.867+0000 7f2284d17700 1 mon.smithi134@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2021-11-19T08:37:15.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.869+0000 7f2284d17700 0 mon.smithi134@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2021-11-19T08:37:15.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.869+0000 7f2284d17700 0 mon.smithi134@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-19T08:37:15.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.869+0000 7f2284d17700 0 mon.smithi134@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-19T08:37:15.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.869+0000 7f2284d17700 0 mon.smithi134@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2021-11-19T08:37:15.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: cluster 2021-11-19T08:37:15.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37: 2021-11-19T08:37:15.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 12.792810+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:37:15.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: .14164) 36 : cluster [DBG] 2021-11-19T08:37:15.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:15.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: cluster 2021-11-19T08:37:15.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:13 2021-11-19T08:37:15.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: .773372+0000 mon.smithi023 (mon. 2021-11-19T08:37:15.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 0) 163 : cluster [DBG] mgrmap e18: smithi023.ehlpcl(active, since 55s), standbys: smithi134.vitoqy 2021-11-19T08:37:15.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: audit 2021-11- 2021-11-19T08:37:15.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 19T08:37:13.773547 2021-11-19T08:37:15.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 2021-11-19T08:37:15.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 164 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mgr metadata", "who": "smithi134.vitoqy", "id": "smithi134.vitoqy"}]: dispatch 2021-11-19T08:37:15.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.870+0000 7f2284d17700 1 mon.smithi134@-1(synchronizing).paxosservice(auth 1..7) refresh upgraded, format 0 -> 3 2021-11-19T08:37:15.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.872+0000 7f2284d17700 20 expand_channel_meta expand map: {default=false} 2021-11-19T08:37:15.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.872+0000 7f2284d17700 20 expand_channel_meta from 'false' to 'false' 2021-11-19T08:37:15.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: 2021-11-19T08:37:15.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.872+0000 7f2284d17700 20 expand_channel_meta expanded map: {default=false} 2021-11-19T08:37:15.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.872+0000 7f2284d17700 20 expand_channel_meta expand map: {default=info} 2021-11-19T08:37:15.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.872+0000 7f2284d17700 20 expand_channel_meta from 'info' to 'info' 2021-11-19T08:37:15.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.872+0000 7f2284d17700 20 expand_channel_meta expanded map: {default=info} 2021-11-19T08:37:15.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.872+0000 7f2284d17700 20 expand_channel_meta expand map: {default=daemon} 2021-11-19T08:37:15.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.872+0000 7f2284d17700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-11-19T08:37:15.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.872+0000 7f2284d17700 20 expand_channel_meta expanded map: {default=daemon} 2021-11-19T08:37:15.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.872+0000 7f2284d17700 20 expand_channel_meta expand map: {default=debug} 2021-11-19T08:37:15.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.872+0000 7f2284d17700 20 expand_channel_meta from 'debug' to 'debug' 2021-11-19T08:37:15.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.872+0000 7f2284d17700 20 expand_channel_meta expanded map: {default=debug} 2021-11-19T08:37:15.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 conmon[36363]: debug 2021-11-19T08:37:15.872+0000 7f2284d17700 10 mon.smithi134@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2021-11-19T08:37:15.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:15 smithi134 systemd[1]: Started Ceph mon.smithi134 for 908b41ee-4913-11ec-8c2c-001a4aab830c. 2021-11-19T08:37:20.898 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.911+0000 7fef621b0700 1 -- 172.21.15.134:0/2049111484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef44003250 msgr2=0x7fef440056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:20.898 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.911+0000 7fef621b0700 1 --2- 172.21.15.134:0/2049111484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef44003250 0x7fef440056e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fef5c0f72c0 tx=0x7fef4c009040).stop 2021-11-19T08:37:20.898 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.911+0000 7fef621b0700 1 -- 172.21.15.134:0/2049111484 shutdown_connections 2021-11-19T08:37:20.899 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.911+0000 7fef621b0700 1 --2- 172.21.15.134:0/2049111484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef44003250 0x7fef440056e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:20.899 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.911+0000 7fef621b0700 1 -- 172.21.15.134:0/2049111484 >> 172.21.15.134:0/2049111484 conn(0x7fef5c0f1c70 msgr2=0x7fef5c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:37:20.899 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.911+0000 7fef621b0700 1 -- 172.21.15.134:0/2049111484 shutdown_connections 2021-11-19T08:37:20.899 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.911+0000 7fef621b0700 1 -- 172.21.15.134:0/2049111484 wait complete. 2021-11-19T08:37:20.900 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.911+0000 7fef621b0700 1 Processor -- start 2021-11-19T08:37:20.900 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.912+0000 7fef621b0700 1 -- start start 2021-11-19T08:37:20.900 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.912+0000 7fef621b0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef44003250 0x7fef5c106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:20.900 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.912+0000 7fef621b0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c106b10 0x7fef5c107a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:20.900 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.912+0000 7fef621b0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c107070 con 0x7fef44003250 2021-11-19T08:37:20.901 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.912+0000 7fef621b0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c1071b0 con 0x7fef5c106b10 2021-11-19T08:37:20.901 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.912+0000 7fef5affd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c106b10 0x7fef5c107a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:20.901 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.912+0000 7fef5affd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c106b10 0x7fef5c107a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47746/0 (socket says 172.21.15.134:47746) 2021-11-19T08:37:20.901 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.912+0000 7fef5affd700 1 -- 172.21.15.134:0/3622486028 learned_addr learned my addr 172.21.15.134:0/3622486028 (peer_addr_for_me v2:172.21.15.134:0/0) 2021-11-19T08:37:20.902 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.913+0000 7fef5affd700 1 -- 172.21.15.134:0/3622486028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c106b10 msgr2=0x7fef5c107a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-11-19T08:37:20.902 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.913+0000 7fef5affd700 1 -- 172.21.15.134:0/3622486028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c106b10 msgr2=0x7fef5c107a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-11-19T08:37:20.902 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.913+0000 7fef5affd700 1 --2- 172.21.15.134:0/3622486028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c106b10 0x7fef5c107a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-19T08:37:20.902 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.913+0000 7fef5b7fe700 1 --2- 172.21.15.134:0/3622486028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef44003250 0x7fef5c106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:20.903 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.913+0000 7fef5affd700 1 --2- 172.21.15.134:0/3622486028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c106b10 0x7fef5c107a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-11-19T08:37:20.904 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.913+0000 7fef5b7fe700 1 -- 172.21.15.134:0/3622486028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c106b10 msgr2=0x7fef5c107a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-11-19T08:37:20.904 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.913+0000 7fef5b7fe700 1 --2- 172.21.15.134:0/3622486028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c106b10 0x7fef5c107a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:20.904 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.913+0000 7fef5b7fe700 1 -- 172.21.15.134:0/3622486028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c0076e0 con 0x7fef44003250 2021-11-19T08:37:20.904 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.913+0000 7fef5b7fe700 1 --2- 172.21.15.134:0/3622486028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef44003250 0x7fef5c106620 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fef4c01f000 tx=0x7fef4c01bf90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:37:20.905 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.913+0000 7fef43fff700 1 -- 172.21.15.134:0/3622486028 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef4c023440 con 0x7fef44003250 2021-11-19T08:37:20.905 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.913+0000 7fef43fff700 1 -- 172.21.15.134:0/3622486028 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef4c00f040 con 0x7fef44003250 2021-11-19T08:37:20.905 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.913+0000 7fef621b0700 1 -- 172.21.15.134:0/3622486028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5c107fb0 con 0x7fef44003250 2021-11-19T08:37:20.906 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.913+0000 7fef43fff700 1 -- 172.21.15.134:0/3622486028 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef4c022850 con 0x7fef44003250 2021-11-19T08:37:20.906 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.914+0000 7fef621b0700 1 -- 172.21.15.134:0/3622486028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5c1084f0 con 0x7fef44003250 2021-11-19T08:37:20.908 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.915+0000 7fef621b0700 1 -- 172.21.15.134:0/3622486028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef5c045ad0 con 0x7fef44003250 2021-11-19T08:37:20.908 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.915+0000 7fef43fff700 1 -- 172.21.15.134:0/3622486028 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fef4c034070 con 0x7fef44003250 2021-11-19T08:37:20.909 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.915+0000 7fef43fff700 1 --2- 172.21.15.134:0/3622486028 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fef4805c400 0x7fef4805e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:20.909 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.916+0000 7fef43fff700 1 -- 172.21.15.134:0/3622486028 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef4c0785a0 con 0x7fef44003250 2021-11-19T08:37:20.909 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.916+0000 7fef5affd700 1 --2- 172.21.15.134:0/3622486028 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fef4805c400 0x7fef4805e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:20.909 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.916+0000 7fef5affd700 1 --2- 172.21.15.134:0/3622486028 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fef4805c400 0x7fef4805e8c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fef50006cf0 tx=0x7fef50008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:37:20.910 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:20.920+0000 7fef43fff700 1 -- 172.21.15.134:0/3622486028 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef4c04fbe0 con 0x7fef44003250 2021-11-19T08:37:21.137 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:21.153+0000 7fef621b0700 1 -- 172.21.15.134:0/3622486028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef5c108760 con 0x7fef44003250 2021-11-19T08:37:21.138 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:21.154+0000 7fef43fff700 1 -- 172.21.15.134:0/3622486028 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+968 (secure 0 0 0) 0x7fef4c04f7d0 con 0x7fef44003250 2021-11-19T08:37:21.139 INFO:teuthology.orchestra.run.smithi134.stdout: 2021-11-19T08:37:21.139 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":2,"fsid":"908b41ee-4913-11ec-8c2c-001a4aab830c","modified":"2021-11-19T08:37:15.875568Z","created":"2021-11-19T08:35:12.680933Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2021-11-19T08:37:21.141 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:21.156+0000 7fef621b0700 1 -- 172.21.15.134:0/3622486028 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fef4805c400 msgr2=0x7fef4805e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:21.141 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:21.156+0000 7fef621b0700 1 --2- 172.21.15.134:0/3622486028 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fef4805c400 0x7fef4805e8c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fef50006cf0 tx=0x7fef50008040).stop 2021-11-19T08:37:21.141 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:21.156+0000 7fef621b0700 1 -- 172.21.15.134:0/3622486028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef44003250 msgr2=0x7fef5c106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:21.142 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:21.156+0000 7fef621b0700 1 --2- 172.21.15.134:0/3622486028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef44003250 0x7fef5c106620 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fef4c01f000 tx=0x7fef4c01bf90).stop 2021-11-19T08:37:21.142 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:21.156+0000 7fef621b0700 1 -- 172.21.15.134:0/3622486028 shutdown_connections 2021-11-19T08:37:21.142 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:21.156+0000 7fef621b0700 1 --2- 172.21.15.134:0/3622486028 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fef4805c400 0x7fef4805e8c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:21.142 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:21.156+0000 7fef621b0700 1 --2- 172.21.15.134:0/3622486028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef44003250 0x7fef5c106620 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:21.143 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:21.156+0000 7fef621b0700 1 --2- 172.21.15.134:0/3622486028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c106b10 0x7fef5c107a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:21.143 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:21.156+0000 7fef621b0700 1 -- 172.21.15.134:0/3622486028 >> 172.21.15.134:0/3622486028 conn(0x7fef5c0f1c70 msgr2=0x7fef5c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:37:21.143 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:21.156+0000 7fef621b0700 1 -- 172.21.15.134:0/3622486028 shutdown_connections 2021-11-19T08:37:21.143 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:37:21.156+0000 7fef621b0700 1 -- 172.21.15.134:0/3622486028 wait complete. 2021-11-19T08:37:21.143 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 2 2021-11-19T08:37:21.166 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: audit 2021-11-19T08:37:15.880793+0000 mon.smithi023 (mon.0) 166 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi023"}]: dispatch 2021-11-19T08:37:21.167 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: audit 2021-11-19T08:37:15.881047+0000 mon.smithi023 (mon.0) 167 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2021-11-19T08:37:21.167 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: cluster 2021-11-19T08:37:15.881200+0000 mon.smithi023 (mon.0) 168 : cluster [INF] mon.smithi023 calling monitor election 2021-11-19T08:37:21.167 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: cluster 2021-11-19T08:37:16.793607+0000 mgr.smithi023.ehlpcl (mgr.14164) 38 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:21.168 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: audit 2021-11-19T08:37:16.875441+0000 mon.smithi023 (mon.0) 169 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2021-11-19T08:37:21.168 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: audit 2021-11-19T08:37:17.875340+0000 mon.smithi023 (mon.0) 170 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2021-11-19T08:37:21.169 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: cluster 2021-11-19T08:37:17.877202+0000 mon.smithi134 (mon.1) 1 : cluster [INF] mon.smithi134 calling monitor election 2021-11-19T08:37:21.169 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: cluster 2021-11-19T08:37:18.794156+0000 mgr.smithi023.ehlpcl (mgr.14164) 39 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:21.170 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: audit 2021-11-19T08:37:18. 2021-11-19T08:37:21.170 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: 875455+0000 mon.smithi023 (mon.0) 171 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2021-11-19T08:37:21.170 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: audit 2021-11-19T08:37:18.967992+0000 mon.smithi023 (mon.0) 172 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:37:21.171 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: audit 2021-11-19T08:37:18.970983+0000 mon.smithi023 (mon.0) 173 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:37:21.171 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: audit 2021-11-19T08:37:19.875429+0000 mon.smithi023 (mon.0) 174 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2021-11-19T08:37:21.172 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: audit 2021-11-19T08:37:20.875643+0000 mon.smithi023 (mon.0) 175 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2021-11-19T08:37:21.172 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: cluster 2021-11-19T08:37:20.896945+0000 mon.smithi023 (mon.0) 176 : cluster [INF] mon.smithi023 is new leader, mons smithi023,smithi134 in quorum (ranks 0,1) 2021-11-19T08:37:21.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: cluster 2021-11-19T08:37:20.900374+0000 mon.smithi023 (mon.0) 177 : cluster [DBG] monmap e2: 2 mons at {smithi023=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],smithi134=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0]} 2021-11-19T08:37:21.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: cluster 2021-11-19T08:37:20.900442+0000 mon.smithi023 (mon.0) 178 : cluster [DBG] fsmap 2021-11-19T08:37:21.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: cluster 2021-11-19T08:37:20.900493+0000 mon.smithi023 (mon.0) 179 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2021-11-19T08:37:21.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: cluster 2021-11-19T08:37:20.900870+0000 mon.smithi023 (mon.0) 180 : cluster [DBG] mgrmap e18: smithi023.ehlpcl(active, since 62s), standbys: smithi134.vitoqy 2021-11-19T08:37:21.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: cluster 2021-11-19T08:37:20.905533+0000 mon.smithi023 (mon.0) 181 : cluster [INF] overall HEALTH_OK 2021-11-19T08:37:21.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:20 smithi134 conmon[36363]: audit 2021-11-19T08:37:20.909385+0000 mon.smithi023 (mon.0) 182 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:21.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: audit 2021-11-19T08:37:15.880793+0000 mon.smithi023 (mon.0 2021-11-19T08:37:21.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: ) 166 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi023"}]: dispatch 2021-11-19T08:37:21.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: audit 2021-11-19T08:37:15.881047+0000 mon.smithi023 (mon.0) 167 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2021-11-19T08:37:21.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: cluster 2021 2021-11-19T08:37:21.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: -11-19T08:37:15.881200+0000 mon.smithi023 (mon.0) 168 : cluster [INF] mon.smithi023 calling monitor election 2021-11-19T08:37:21.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: cluster 2021-11- 2021-11-19T08:37:21.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: 19T08:37:16.793607+0000 mgr.smithi023.ehlpcl (mgr.14164) 38 : cluster 2021-11-19T08:37:21.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:21.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:37:21.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: T08:37:16.875441+0000 mon.smithi023 (mon.0) 169 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2021-11-19T08:37:21.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:37:21.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: -19T08:37:17.875340+0000 2021-11-19T08:37:21.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: mon.smithi023 (mon.0) 170 : audit [DBG] 2021-11-19T08:37:21.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2021-11-19T08:37:21.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: cluster 2021-11-19T08:37:17 2021-11-19T08:37:21.398 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: .877202+0000 mon.smithi134 (mon.1) 1 : 2021-11-19T08:37:21.398 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: cluster [INF] mon.smithi134 calling monitor election 2021-11-19T08:37:21.398 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:37:21.398 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: 08:37:18.794156+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:37:21.399 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: .14164) 39 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:21.399 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:37:21.399 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: 11-19T08:37:18.875455+0000 2021-11-19T08:37:21.399 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: mon.smithi023 (mon.0) 171 : audit [DBG] 2021-11-19T08:37:21.400 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2021-11-19T08:37:21.400 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: audit 2021-11-19T08:37 2021-11-19T08:37:21.400 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: :18.967992+0000 mon.smithi023 (mon.0) 2021-11-19T08:37:21.400 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: 172 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:37:21.400 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: audit 2021-11- 2021-11-19T08:37:21.401 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: 19T08:37:18.970983+0000 mon.smithi023 ( 2021-11-19T08:37:21.401 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: mon.0) 173 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:37:21.401 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: 2021-11-19T08:37:21.401 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: audit 2021-11-19T08:37:21.402 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: 2021-11-19T08:37:19.875429+0000 2021-11-19T08:37:21.402 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: mon.smithi023 (mon.0) 174 : audit [DBG] 2021-11-19T08:37:21.402 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2021-11-19T08:37:21.402 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: audit 2021-11-19T08:37:21.402 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: 2021-11-19T08:37:20.875643+0000 2021-11-19T08:37:21.403 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: mon.smithi023 (mon.0) 175 : audit [DBG] 2021-11-19T08:37:21.403 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2021-11-19T08:37:21.403 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: cluster 2021-11-19T08:37 2021-11-19T08:37:21.403 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: :20.896945+0000 mon.smithi023 (mon.0 2021-11-19T08:37:21.404 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: ) 176 : cluster [INF] mon.smithi023 is new leader, mons smithi023,smithi134 in quorum (ranks 0,1) 2021-11-19T08:37:21.404 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: cluster 2021-11- 2021-11-19T08:37:21.404 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: 19T08:37:20.900374+0000 mon.smithi023 2021-11-19T08:37:21.404 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: (mon.0) 177 : cluster [DBG] monmap e2: 2 mons at {smithi023=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],smithi134=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0]} 2021-11-19T08:37:21.404 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: 2021-11-19T08:37:21.405 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: cluster 2021-11-19T08:37:21.405 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: 2021-11-19T08:37:20.900442+0000 mon.smithi023 (mon.0) 178 : 2021-11-19T08:37:21.405 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: cluster [DBG] fsmap 2021-11-19T08:37:21.405 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: cluster 2021-11 2021-11-19T08:37:21.406 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: -19T08:37:20.900493+0000 2021-11-19T08:37:21.406 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: mon.smithi023 (mon.0) 179 : cluster [DBG] 2021-11-19T08:37:21.406 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: osdmap e5: 0 total, 0 up, 0 in 2021-11-19T08:37:21.406 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: cluster 2021-11-19T08:37:21.406 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: 2021-11-19T08:37:20.900870+0000 2021-11-19T08:37:21.407 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: mon.smithi023 (mon.0) 180 : cluster [DBG] mgrmap e18: smithi023.ehlpcl(active, since 62s), standbys: smithi134.vitoqy 2021-11-19T08:37:21.407 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: 2021-11-19T08:37:21.407 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: cluster 2021-11-19T08:37:21.408 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: 2021-11-19T08:37:20.905533+0000 mon.smithi023 2021-11-19T08:37:21.408 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: (mon.0) 181 : cluster [INF] overall HEALTH_OK 2021-11-19T08:37:21.409 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: audit 2021 2021-11-19T08:37:21.409 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: -11-19T08:37:20.909385 2021-11-19T08:37:21.409 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 182 : audit 2021-11-19T08:37:21.409 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:20 smithi023 conmon[33490]: [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:21.794 INFO:tasks.cephadm:Generating final ceph.conf file... 2021-11-19T08:37:21.795 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph config generate-minimal-conf 2021-11-19T08:37:22.130 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:21 smithi134 conmon[36363]: cluster 2021-11-19T08:37:20.794578+0000 mgr.smithi023.ehlpcl (mgr.14164) 40 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:22.130 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:21 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:20.910611+0000 mgr.smithi023.ehlpcl (mgr.14164) 41 : cephadm [INF] Deploying daemon node-exporter.smithi134 on smithi134 2021-11-19T08:37:22.131 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:21 smithi134 conmon[36363]: audit 2021-11-19T08:37:21.153755+0000 mon.smithi023 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.134:0/3622486028' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-19T08:37:22.131 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:21 smithi134 conmon[36363]: audit 2021-11-19T08:37:21.875882+0000 mon.smithi023 (mon.0) 184 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2021-11-19T08:37:22.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:21 smithi023 conmon[33490]: cluster 2021-11-19T08:37:20.794578+0000 mgr.smithi023.ehlpcl 2021-11-19T08:37:22.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:21 smithi023 conmon[33490]: (mgr.14164) 40 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:22.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:21 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:20.910611+0000 mgr.smithi023.ehlpcl (mgr.14164) 41 : cephadm [INF] Deploying daemon node-exporter.smithi134 on smithi134 2021-11-19T08:37:22.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:21 smithi023 conmon[33490]: audit 2021-11-19T08:37:21.153755+0000 mon.smithi023 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.134:0/3622486028' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-11-19T08:37:22.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:21 smithi023 conmon[33490]: audit 2021-11-19T08:37:21.875882+0000 mon.smithi023 (mon.0) 184 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2021-11-19T08:37:23.470 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.483+0000 7efc67a1a700 1 -- 172.21.15.23:0/2446409211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc600f5410 msgr2=0x7efc600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:23.470 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.483+0000 7efc67a1a700 1 --2- 172.21.15.23:0/2446409211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc600f5410 0x7efc600f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7efc5c009fb0 tx=0x7efc5c009b30).stop 2021-11-19T08:37:23.471 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.483+0000 7efc67a1a700 1 -- 172.21.15.23:0/2446409211 shutdown_connections 2021-11-19T08:37:23.471 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.483+0000 7efc67a1a700 1 --2- 172.21.15.23:0/2446409211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc600f5410 0x7efc600f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:23.471 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.483+0000 7efc67a1a700 1 -- 172.21.15.23:0/2446409211 >> 172.21.15.23:0/2446409211 conn(0x7efc600f0b70 msgr2=0x7efc600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:37:23.471 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.483+0000 7efc67a1a700 1 -- 172.21.15.23:0/2446409211 shutdown_connections 2021-11-19T08:37:23.472 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.483+0000 7efc67a1a700 1 -- 172.21.15.23:0/2446409211 wait complete. 2021-11-19T08:37:23.472 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.483+0000 7efc67a1a700 1 Processor -- start 2021-11-19T08:37:23.472 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.483+0000 7efc67a1a700 1 -- start start 2021-11-19T08:37:23.472 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc67a1a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc600f5410 0x7efc601094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:23.473 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc67a1a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc60060220 0x7efc6010b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:23.473 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc67a1a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6010bf90 con 0x7efc600f5410 2021-11-19T08:37:23.473 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc67a1a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6010c0d0 con 0x7efc60060220 2021-11-19T08:37:23.473 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc657b6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc600f5410 0x7efc601094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:23.474 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc657b6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc600f5410 0x7efc601094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46380/0 (socket says 172.21.15.23:46380) 2021-11-19T08:37:23.474 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc657b6700 1 -- 172.21.15.23:0/1689840884 learned_addr learned my addr 172.21.15.23:0/1689840884 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:37:23.474 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc64fb5700 1 --2- 172.21.15.23:0/1689840884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc60060220 0x7efc6010b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:23.474 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc657b6700 1 -- 172.21.15.23:0/1689840884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc60060220 msgr2=0x7efc6010b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:23.474 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc657b6700 1 --2- 172.21.15.23:0/1689840884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc60060220 0x7efc6010b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:23.475 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc657b6700 1 -- 172.21.15.23:0/1689840884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5c005040 con 0x7efc600f5410 2021-11-19T08:37:23.475 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc64fb5700 1 --2- 172.21.15.23:0/1689840884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc60060220 0x7efc6010b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:37:23.475 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc657b6700 1 --2- 172.21.15.23:0/1689840884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc600f5410 0x7efc601094c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7efc5c002040 tx=0x7efc5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:37:23.475 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc55ffb700 1 -- 172.21.15.23:0/1689840884 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc5c019070 con 0x7efc600f5410 2021-11-19T08:37:23.476 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc67a1a700 1 -- 172.21.15.23:0/1689840884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc601052a0 con 0x7efc600f5410 2021-11-19T08:37:23.476 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc55ffb700 1 -- 172.21.15.23:0/1689840884 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc5c007c70 con 0x7efc600f5410 2021-11-19T08:37:23.476 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc55ffb700 1 -- 172.21.15.23:0/1689840884 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc5c01d620 con 0x7efc600f5410 2021-11-19T08:37:23.476 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.484+0000 7efc67a1a700 1 -- 172.21.15.23:0/1689840884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc60105800 con 0x7efc600f5410 2021-11-19T08:37:23.477 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.485+0000 7efc67a1a700 1 -- 172.21.15.23:0/1689840884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc60045ad0 con 0x7efc600f5410 2021-11-19T08:37:23.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.486+0000 7efc55ffb700 1 -- 172.21.15.23:0/1689840884 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efc5c025070 con 0x7efc600f5410 2021-11-19T08:37:23.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.486+0000 7efc55ffb700 1 --2- 172.21.15.23:0/1689840884 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7efc4c05c3b0 0x7efc4c05e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:23.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.486+0000 7efc55ffb700 1 -- 172.21.15.23:0/1689840884 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc5c076260 con 0x7efc600f5410 2021-11-19T08:37:23.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.486+0000 7efc64fb5700 1 --2- 172.21.15.23:0/1689840884 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7efc4c05c3b0 0x7efc4c05e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:23.479 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.487+0000 7efc64fb5700 1 --2- 172.21.15.23:0/1689840884 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7efc4c05c3b0 0x7efc4c05e870 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7efc6005f9a0 tx=0x7efc50008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:37:23.479 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.490+0000 7efc55ffb700 1 -- 172.21.15.23:0/1689840884 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc5c049f30 con 0x7efc600f5410 2021-11-19T08:37:23.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.647+0000 7efc67a1a700 1 -- 172.21.15.23:0/1689840884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7efc60026880 con 0x7efc600f5410 2021-11-19T08:37:23.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.648+0000 7efc55ffb700 1 -- 172.21.15.23:0/1689840884 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v12) v1 ==== 76+0+225 (secure 0 0 0) 0x7efc5c04d550 con 0x7efc600f5410 2021-11-19T08:37:23.635 INFO:teuthology.orchestra.run.smithi023.stdout:# minimal ceph.conf for 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:37:23.635 INFO:teuthology.orchestra.run.smithi023.stdout:[global] 2021-11-19T08:37:23.635 INFO:teuthology.orchestra.run.smithi023.stdout: fsid = 908b41ee-4913-11ec-8c2c-001a4aab830c 2021-11-19T08:37:23.635 INFO:teuthology.orchestra.run.smithi023.stdout: mon_host = [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2021-11-19T08:37:23.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.649+0000 7efc67a1a700 1 -- 172.21.15.23:0/1689840884 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7efc4c05c3b0 msgr2=0x7efc4c05e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:23.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.649+0000 7efc67a1a700 1 --2- 172.21.15.23:0/1689840884 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7efc4c05c3b0 0x7efc4c05e870 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7efc6005f9a0 tx=0x7efc50008040).stop 2021-11-19T08:37:23.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.649+0000 7efc67a1a700 1 -- 172.21.15.23:0/1689840884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc600f5410 msgr2=0x7efc601094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:23.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.649+0000 7efc67a1a700 1 --2- 172.21.15.23:0/1689840884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc600f5410 0x7efc601094c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7efc5c002040 tx=0x7efc5c00b040).stop 2021-11-19T08:37:23.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.649+0000 7efc67a1a700 1 -- 172.21.15.23:0/1689840884 shutdown_connections 2021-11-19T08:37:23.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.649+0000 7efc67a1a700 1 --2- 172.21.15.23:0/1689840884 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7efc4c05c3b0 0x7efc4c05e870 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.649+0000 7efc67a1a700 1 --2- 172.21.15.23:0/1689840884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc600f5410 0x7efc601094c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.649+0000 7efc67a1a700 1 --2- 172.21.15.23:0/1689840884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc60060220 0x7efc6010b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.649+0000 7efc67a1a700 1 -- 172.21.15.23:0/1689840884 >> 172.21.15.23:0/1689840884 conn(0x7efc600f0b70 msgr2=0x7efc600f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:37:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.649+0000 7efc67a1a700 1 -- 172.21.15.23:0/1689840884 shutdown_connections 2021-11-19T08:37:23.639 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:23.650+0000 7efc67a1a700 1 -- 172.21.15.23:0/1689840884 wait complete. 2021-11-19T08:37:24.040 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2021-11-19T08:37:24.040 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2021-11-19T08:37:24.041 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.conf 2021-11-19T08:37:24.070 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2021-11-19T08:37:24.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-19T08:37:24.137 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2021-11-19T08:37:24.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.conf 2021-11-19T08:37:24.163 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:23 smithi134 conmon[36363]: cluster 2021-11-19T08:37:22.794815+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:37:24.164 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:23 smithi134 conmon[36363]: ) 42 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:24.164 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:23 smithi134 conmon[36363]: audit 2021-11-19T08:37:23.648763+0000 mon.smithi023 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.23:0/1689840884' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:24.168 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2021-11-19T08:37:24.168 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-11-19T08:37:24.234 INFO:tasks.cephadm:Deploying OSDs... 2021-11-19T08:37:24.235 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2021-11-19T08:37:24.235 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2021-11-19T08:37:24.252 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-11-19T08:37:24.252 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-11-19T08:37:24.307 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-11-19T08:37:24.308 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-19T08:37:24.309 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 6h/6d Inode: 116121 Links: 1 2021-11-19T08:37:24.309 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-19T08:37:24.309 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2021-11-19T08:37:24.309 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2021-11-19 08:36:58.173440784 +0000 2021-11-19T08:37:24.310 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2021-11-19 08:30:41.987473014 +0000 2021-11-19T08:37:24.310 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2021-11-19 08:30:41.987473014 +0000 2021-11-19T08:37:24.310 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2021-11-19T08:37:24.315 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:23 smithi023 conmon[33490]: cluster 2021-11-19T08:37:22.794815+0000 mgr.smithi023.ehlpcl (mgr.14164) 42 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:24.316 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:23 smithi023 conmon[33490]: audit 2021-11-19T08:37:23.648763+0000 mon.smithi023 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.23:0/1689840884' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:24.318 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2021-11-19T08:37:24.318 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2021-11-19T08:37:24.318 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000239784 s, 2.1 MB/s 2021-11-19T08:37:24.321 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-11-19T08:37:24.377 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-11-19T08:37:24.377 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-19T08:37:24.378 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 6h/6d Inode: 117128 Links: 1 2021-11-19T08:37:24.378 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-19T08:37:24.378 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2021-11-19T08:37:24.378 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2021-11-19 08:36:58.173440784 +0000 2021-11-19T08:37:24.379 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2021-11-19 08:30:42.345470474 +0000 2021-11-19T08:37:24.379 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2021-11-19 08:30:42.345470474 +0000 2021-11-19T08:37:24.379 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2021-11-19T08:37:24.388 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2021-11-19T08:37:24.388 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2021-11-19T08:37:24.389 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000227176 s, 2.3 MB/s 2021-11-19T08:37:24.391 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-11-19T08:37:24.448 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-11-19T08:37:24.448 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-19T08:37:24.448 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 6h/6d Inode: 118130 Links: 1 2021-11-19T08:37:24.449 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-19T08:37:24.449 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2021-11-19T08:37:24.449 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2021-11-19 08:36:58.173440784 +0000 2021-11-19T08:37:24.449 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2021-11-19 08:30:42.688468041 +0000 2021-11-19T08:37:24.449 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2021-11-19 08:30:42.688468041 +0000 2021-11-19T08:37:24.450 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2021-11-19T08:37:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2021-11-19T08:37:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2021-11-19T08:37:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000242326 s, 2.1 MB/s 2021-11-19T08:37:24.462 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-11-19T08:37:24.518 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-11-19T08:37:24.518 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-19T08:37:24.519 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 6h/6d Inode: 116204 Links: 1 2021-11-19T08:37:24.519 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-19T08:37:24.519 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2021-11-19T08:37:24.519 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2021-11-19 08:36:58.174440778 +0000 2021-11-19T08:37:24.519 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2021-11-19 08:30:43.038465558 +0000 2021-11-19T08:37:24.520 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2021-11-19 08:30:43.038465558 +0000 2021-11-19T08:37:24.520 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2021-11-19T08:37:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2021-11-19T08:37:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2021-11-19T08:37:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000237509 s, 2.2 MB/s 2021-11-19T08:37:24.531 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2021-11-19T08:37:24.531 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2021-11-19T08:37:24.551 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-11-19T08:37:24.552 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-11-19T08:37:24.607 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-11-19T08:37:24.607 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-19T08:37:24.608 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 6h/6d Inode: 118889 Links: 1 2021-11-19T08:37:24.608 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-19T08:37:24.608 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2021-11-19T08:37:24.609 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2021-11-19 08:37:21.947119884 +0000 2021-11-19T08:37:24.609 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2021-11-19 08:30:33.355860710 +0000 2021-11-19T08:37:24.609 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2021-11-19 08:30:33.355860710 +0000 2021-11-19T08:37:24.609 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2021-11-19T08:37:24.617 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2021-11-19T08:37:24.617 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2021-11-19T08:37:24.618 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000212142 s, 2.4 MB/s 2021-11-19T08:37:24.620 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-11-19T08:37:24.676 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-11-19T08:37:24.676 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-19T08:37:24.676 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 6h/6d Inode: 114274 Links: 1 2021-11-19T08:37:24.676 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-19T08:37:24.677 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2021-11-19T08:37:24.677 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2021-11-19 08:37:21.947119884 +0000 2021-11-19T08:37:24.677 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2021-11-19 08:30:33.688852963 +0000 2021-11-19T08:37:24.677 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2021-11-19 08:30:33.688852963 +0000 2021-11-19T08:37:24.678 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2021-11-19T08:37:24.686 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2021-11-19T08:37:24.686 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2021-11-19T08:37:24.687 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000208917 s, 2.5 MB/s 2021-11-19T08:37:24.689 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-11-19T08:37:24.750 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-11-19T08:37:24.750 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-19T08:37:24.751 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 6h/6d Inode: 114331 Links: 1 2021-11-19T08:37:24.751 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-19T08:37:24.751 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2021-11-19T08:37:24.751 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2021-11-19 08:37:21.947119884 +0000 2021-11-19T08:37:24.752 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2021-11-19 08:30:34.039844796 +0000 2021-11-19T08:37:24.752 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2021-11-19 08:30:34.039844796 +0000 2021-11-19T08:37:24.752 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2021-11-19T08:37:24.758 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2021-11-19T08:37:24.758 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2021-11-19T08:37:24.759 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000247056 s, 2.1 MB/s 2021-11-19T08:37:24.761 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-11-19T08:37:24.821 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-11-19T08:37:24.822 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-11-19T08:37:24.822 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 6h/6d Inode: 114402 Links: 1 2021-11-19T08:37:24.822 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-11-19T08:37:24.822 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2021-11-19T08:37:24.823 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2021-11-19 08:37:21.947119884 +0000 2021-11-19T08:37:24.823 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2021-11-19 08:30:34.390836630 +0000 2021-11-19T08:37:24.823 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2021-11-19 08:30:34.390836630 +0000 2021-11-19T08:37:24.823 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2021-11-19T08:37:24.827 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2021-11-19T08:37:24.827 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2021-11-19T08:37:24.828 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000191844 s, 2.7 MB/s 2021-11-19T08:37:24.830 INFO:tasks.cephadm:Deploying osd.0 on smithi023 with /dev/vg_nvme/lv_4... 2021-11-19T08:37:24.831 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-11-19T08:37:25.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:25 smithi134 conmon[36363]: audit 2021-11-19T08:37:24.271663+0000 mon.smithi023 (mon.0) 186 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:25.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:25 smithi134 conmon[36363]: audit 2021-11-19T08:37:24.272496+0000 mon.smithi023 (mon.0) 187 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:37:25.612 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:25 smithi023 conmon[33490]: audit 2021-11-19T08:37:24.271663+0000 mon.smithi023 (mon.0) 186 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:25.613 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:25 smithi023 conmon[33490]: audit 2021-11-19T08:37:24.272496+0000 mon.smithi023 (mon.0) 187 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:37:26.528 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:26 smithi023 conmon[33490]: cluster 2021-11-19T08:37:24.795058+0000 mgr.smithi023.ehlpcl (mgr.14164) 43 : 2021-11-19T08:37:26.529 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:26 smithi023 conmon[33490]: cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:26.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:26 smithi134 conmon[36363]: cluster 2021-11-19T08:37:24.795058+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:37:26.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:26 smithi134 conmon[36363]: 43 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:27.120 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-11-19T08:37:27.120 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-11-19T08:37:27.121 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2021-11-19T08:37:27.121 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2021-11-19T08:37:27.121 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0155132 s, 676 MB/s 2021-11-19T08:37:27.121 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2021-11-19T08:37:27.553 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi023:vg_nvme/lv_4 2021-11-19T08:37:27.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:27 smithi134 conmon[36363]: audit 2021-11-19T08:37:26.418925+0000 mon.smithi023 (mon.0) 188 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:27.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:27 smithi023 conmon[33490]: audit 2021-11-19T08:37:26.418925+0000 2021-11-19T08:37:27.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:27 smithi023 conmon[33490]: mon.smithi023 (mon.0) 188 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:28.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:28 smithi023 conmon[33490]: cluster 2021-11-19T08:37: 2021-11-19T08:37:28.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:28 smithi023 conmon[33490]: 26.795262+0000 mgr.smithi023.ehlpcl (mgr.14164) 44 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:28.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:28 smithi134 conmon[36363]: cluster 2021-11-19T08:37:26.795262+0000 mgr.smithi023.ehlpcl (mgr.14164) 44 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:30.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.267+0000 7f392389b700 1 -- 172.21.15.23:0/940860445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f391c0fbc50 msgr2=0x7f391c0fe040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:30.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.267+0000 7f392389b700 1 --2- 172.21.15.23:0/940860445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f391c0fbc50 0x7f391c0fe040 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f390c004690 tx=0x7f390c009b30).stop 2021-11-19T08:37:30.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.268+0000 7f392389b700 1 -- 172.21.15.23:0/940860445 shutdown_connections 2021-11-19T08:37:30.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.268+0000 7f392389b700 1 --2- 172.21.15.23:0/940860445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f391c0fbc50 0x7f391c0fe040 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:30.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.268+0000 7f392389b700 1 --2- 172.21.15.23:0/940860445 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f391c060390 0x7f391c0fb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-19T08:37:30.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.268+0000 7f392389b700 1 -- 172.21.15.23:0/940860445 >> 172.21.15.23:0/940860445 conn(0x7f391c0f0de0 msgr2=0x7f391c0f3200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:37:30.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.268+0000 7f392389b700 1 -- 172.21.15.23:0/940860445 shutdown_connections 2021-11-19T08:37:30.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.268+0000 7f392389b700 1 -- 172.21.15.23:0/940860445 wait complete. 2021-11-19T08:37:30.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.268+0000 7f392389b700 1 Processor -- start 2021-11-19T08:37:30.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.268+0000 7f392389b700 1 -- start start 2021-11-19T08:37:30.258 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.268+0000 7f392389b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f391c060390 0x7f391c1091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:30.258 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.268+0000 7f392389b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f391c0fbc50 0x7f391c1096d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:30.258 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.268+0000 7f392389b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391c109d10 con 0x7f391c0fbc50 2021-11-19T08:37:30.258 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.268+0000 7f392389b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391c10a5c0 con 0x7f391c060390 2021-11-19T08:37:30.259 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.269+0000 7f3920e36700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f391c0fbc50 0x7f391c1096d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:30.259 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.269+0000 7f3920e36700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f391c0fbc50 0x7f391c1096d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46392/0 (socket says 172.21.15.23:46392) 2021-11-19T08:37:30.259 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.269+0000 7f3920e36700 1 -- 172.21.15.23:0/870336013 learned_addr learned my addr 172.21.15.23:0/870336013 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:37:30.259 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.269+0000 7f3921637700 1 --2- 172.21.15.23:0/870336013 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f391c060390 0x7f391c1091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:30.260 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.269+0000 7f3920e36700 1 -- 172.21.15.23:0/870336013 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f391c060390 msgr2=0x7f391c1091e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:30.260 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.269+0000 7f3920e36700 1 --2- 172.21.15.23:0/870336013 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f391c060390 0x7f391c1091e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:30.260 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.269+0000 7f3920e36700 1 -- 172.21.15.23:0/870336013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390c005040 con 0x7f391c0fbc50 2021-11-19T08:37:30.260 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.269+0000 7f3920e36700 1 --2- 172.21.15.23:0/870336013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f391c0fbc50 0x7f391c1096d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f390c004e20 tx=0x7f390c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:37:30.260 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.269+0000 7f3911ffb700 1 -- 172.21.15.23:0/870336013 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f390c019070 con 0x7f391c0fbc50 2021-11-19T08:37:30.261 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.269+0000 7f392389b700 1 -- 172.21.15.23:0/870336013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f391c10a840 con 0x7f391c0fbc50 2021-11-19T08:37:30.261 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.269+0000 7f3911ffb700 1 -- 172.21.15.23:0/870336013 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f390c007c70 con 0x7f391c0fbc50 2021-11-19T08:37:30.261 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.269+0000 7f3911ffb700 1 -- 172.21.15.23:0/870336013 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f390c01d620 con 0x7f391c0fbc50 2021-11-19T08:37:30.261 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.269+0000 7f392389b700 1 -- 172.21.15.23:0/870336013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f391c10ad20 con 0x7f391c0fbc50 2021-11-19T08:37:30.262 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.271+0000 7f3911ffb700 1 -- 172.21.15.23:0/870336013 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f390c025070 con 0x7f391c0fbc50 2021-11-19T08:37:30.263 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.271+0000 7f392389b700 1 -- 172.21.15.23:0/870336013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391c0f4a70 con 0x7f391c0fbc50 2021-11-19T08:37:30.263 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.271+0000 7f3911ffb700 1 --2- 172.21.15.23:0/870336013 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f390805c2b0 0x7f390805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:30.263 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.271+0000 7f3911ffb700 1 -- 172.21.15.23:0/870336013 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f390c0762a0 con 0x7f391c0fbc50 2021-11-19T08:37:30.263 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.271+0000 7f3921637700 1 --2- 172.21.15.23:0/870336013 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f390805c2b0 0x7f390805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:30.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.272+0000 7f3921637700 1 --2- 172.21.15.23:0/870336013 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f390805c2b0 0x7f390805e770 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3918006fd0 tx=0x7f3918008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:37:30.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.275+0000 7f3911ffb700 1 -- 172.21.15.23:0/870336013 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f390c049ef0 con 0x7f391c0fbc50 2021-11-19T08:37:30.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:30.476+0000 7f392389b700 1 -- 172.21.15.23:0/870336013 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f391c101140 con 0x7f390805c2b0 2021-11-19T08:37:30.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:30 smithi134 conmon[36363]: cluster 2021-11-19T08:37:28.795714+0000 mgr.smithi023.ehlpcl (mgr.14164) 45 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:30.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:30 smithi023 conmon[33490]: cluster 2021-11-19T08:37:28.795714+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:37:30.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:30 smithi023 conmon[33490]: ) 45 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:31.693 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:31 smithi023 conmon[33490]: audit 2021-11-19T08:37:30.478349+0000 mgr.smithi023.ehlpcl (mgr.14164) 46 : audit [DBG] from='client.14212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:37:31.694 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:31 smithi023 conmon[33490]: audit 2021-11-19T08:37:30.479184+0000 mon.smithi023 (mon.0) 189 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-19T08:37:31.694 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:31 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:30.480588+0000 mgr.smithi023.ehlpcl (mgr.14164) 47 : cephadm [INF] Applying service osd.None on host smithi023... 2021-11-19T08:37:31.694 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:31 smithi023 conmon[33490]: audit 2021-11-19T08:37:30.480813+0000 mon.smithi023 (mon.0) 190 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-19T08:37:31.695 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:31 smithi023 conmon[33490]: audit 2021-11-19T08:37:30.481469+0000 mon.smithi023 (mon.0) 191 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:31.695 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:31 smithi023 conmon[33490]: audit 2021-11-19T08:37:30.766845+0000 mon.smithi023 (mon.0) 192 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:31.695 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:31 smithi023 conmon[33490]: audit 2021 2021-11-19T08:37:31.695 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:31 smithi023 conmon[33490]: -11-19T08:37:31.185766+0000 mon.smithi023 (mon.0) 193 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:31.695 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:31 smithi023 conmon[33490]: audit 2021-11-19T08:37:31.188244+0000 mon.smithi023 (mon.0) 194 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-19T08:37:31.696 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:31 smithi023 conmon[33490]: audit 2021-11-19T08:37:31.188983+0000 mon.smithi023 (mon.0) 195 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-19T08:37:31.696 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:31 smithi023 conmon[33490]: audit 2021-11-19T08:37:31.189607+0000 mon.smithi023 (mon.0) 196 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:31.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: audit 2021-11-19T08:37:30.478349+0000 mgr.smithi023.ehlpcl (mgr.14164) 46 : audit [DBG] from='client.14212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:37:31.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: audit 2021-11-19T08:37:30.479184+0000 mon.smithi023 (mon.0) 189 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-19T08:37:31.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: cephadm 2021 2021-11-19T08:37:31.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: -11-19T08:37:30.480588+0000 mgr.smithi023.ehlpcl (mgr.14164) 47 : cephadm [INF] Applying service osd.None on host smithi023... 2021-11-19T08:37:31.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: 2021-11-19T08:37:31.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: audit 2021-11-19T08:37:31.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: 2021-11-19T08:37:30.480813+0000 mon.smithi023 2021-11-19T08:37:31.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: (mon.0) 190 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-19T08:37:31.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:37:31.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: 11-19T08:37:30.481469+0000 2021-11-19T08:37:31.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: mon.smithi023 (mon.0) 191 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:31.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: 2021-11-19T08:37:31.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: audit 2021-11-19T08:37:31.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: 2021-11-19T08:37:30.766845+0000 mon.smithi023 2021-11-19T08:37:31.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: (mon.0) 192 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:31.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:37:31.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: 11-19T08:37:31.185766+0000 2021-11-19T08:37:31.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: mon.smithi023 (mon.0) 193 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:31.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: 2021-11-19T08:37:31.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: audit 2021-11-19T08:37:31.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: 2021-11-19T08:37:31.188244+0000 2021-11-19T08:37:31.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: mon.smithi023 (mon.0) 194 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-19T08:37:31.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: audit 2021-11 2021-11-19T08:37:31.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: -19T08:37:31.188983+0000 mon.smithi023 ( 2021-11-19T08:37:31.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: mon.0) 195 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-19T08:37:31.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:37:31.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: 11-19T08:37:31.189607+0000 2021-11-19T08:37:31.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: mon.smithi023 (mon.0) 196 : audit [DBG] 2021-11-19T08:37:31.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:31 smithi134 conmon[36363]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:33.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: cluster 2021-11-19T08:37:30.795975+0000 mgr.smithi023.ehlpcl (mgr.14164) 48 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:33.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:31.187823+0000 mgr.smithi023.ehlpcl (mgr.14164) 49 : cephadm [INF] Reconfiguring mon.smithi023 (unknown last config time)... 2021-11-19T08:37:33.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: cephadm 2021-11-19T08:37: 2021-11-19T08:37:33.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: 31.190209+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:37:33.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: 50 : cephadm [INF] Reconfiguring daemon mon.smithi023 on smithi023 2021-11-19T08:37:33.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: audit 2021-11 2021-11-19T08:37:33.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: -19T08:37:32.693045+0000 mon.smithi023 2021-11-19T08:37:33.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: (mon.0) 197 : audit [INF] 2021-11-19T08:37:33.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: from='client.? 172.21.15.23:0/690886504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55d41e72-4389-4078-ab49-915578ae7162"}]: dispatch 2021-11-19T08:37:33.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: audit 2021 2021-11-19T08:37:33.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: -11-19T08:37:32. 2021-11-19T08:37:33.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: 697827+0000 mon.smithi023 (mon.0) 2021-11-19T08:37:33.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: 198 : audit [INF] from='client.? 172.21.15.23:0/690886504' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55d41e72-4389-4078-ab49-915578ae7162"}]': finished 2021-11-19T08:37:33.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: cluster 2021 2021-11-19T08:37:33.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: -11-19T08:37:32. 2021-11-19T08:37:33.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: 697902+0000 mon.smithi023 (mon.0) 2021-11-19T08:37:33.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: 199 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-11-19T08:37:33.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: audit 2021 2021-11-19T08:37:33.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: -11-19T08:37:32. 2021-11-19T08:37:33.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: 698033+0000 mon.smithi023 (mon.0) 2021-11-19T08:37:33.046 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:32 smithi134 conmon[36363]: 200 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-19T08:37:33.107 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:32 smithi023 conmon[33490]: cluster 2021-11-19T08:37:30.795975+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:37:33.107 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:32 smithi023 conmon[33490]: 48 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:33.108 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:32 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:31.187823+0000 mgr.smithi023.ehlpcl (mgr.14164) 49 : cephadm [INF] Reconfiguring mon.smithi023 (unknown last config time)... 2021-11-19T08:37:33.108 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:32 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:31.190209+0000 mgr.smithi023.ehlpcl (mgr.14164) 50 : cephadm [INF] Reconfiguring daemon mon.smithi023 on smithi023 2021-11-19T08:37:33.109 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:32 smithi023 conmon[33490]: audit 2021-11-19T08:37:32.693045+0000 mon.smithi023 (mon.0) 197 : audit [INF] from='client.? 172.21.15.23:0/690886504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55d41e72-4389-4078-ab49-915578ae7162"}]: dispatch 2021-11-19T08:37:33.110 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:32 smithi023 conmon[33490]: audit 2021-11-19T08:37:32.697827+0000 mon.smithi023 (mon.0) 198 : audit [INF] from='client.? 172.21.15.23:0/690886504' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55d41e72-4389-4078-ab49-915578ae7162"}]': finished 2021-11-19T08:37:33.110 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:32 smithi023 conmon[33490]: cluster 2021-11-19T08:37:32.697902+0000 mon.smithi023 (mon.0) 199 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-11-19T08:37:33.111 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:32 smithi023 conmon[33490]: audit 2021-11-19T08:37:32.698033+0000 mon.smithi023 (mon.0) 200 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-19T08:37:34.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:34 smithi023 conmon[33490]: audit 2021-11-19T08:37:33.121066+0000 mon.smithi023 (mon 2021-11-19T08:37:34.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:34 smithi023 conmon[33490]: .0) 201 : audit [DBG] from='client.? 172.21.15.23:0/3861101354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-19T08:37:34.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:34 smithi134 conmon[36363]: audit 2021-11-19T08:37:33.121066+0000 mon.smithi023 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.23:0/3861101354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-19T08:37:35.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:35 smithi023 conmon[33490]: cluster 2021-11-19T08:37:32.796222+0000 mgr.smithi023.ehlpcl (mgr.14164) 51 2021-11-19T08:37:35.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:35 smithi023 conmon[33490]: : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:35.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:35 smithi134 conmon[36363]: cluster 2021-11-19T08:37:32.796222+0000 mgr.smithi023.ehlpcl (mgr.14164) 51 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:36.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:36 smithi023 conmon[33490]: cluster 2021-11-19T08:37:34.796574+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:37:36.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:36 smithi023 conmon[33490]: 52 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:36.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:36 smithi134 conmon[36363]: cluster 2021-11-19T08:37:34.796574+0000 mgr.smithi023.ehlpcl (mgr.14164) 52 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:38.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:38 smithi023 conmon[33490]: cluster 2021-11-19T08:37:36.796964+0000 mgr.smithi023.ehlpcl (mgr.14164) 53 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:38.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:38 smithi134 conmon[36363]: cluster 2021-11-19T08:37:36.796964+0000 mgr.smithi023.ehlpcl (mgr.14164) 53 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:39.973 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:39 smithi023 conmon[33490]: audit 2021-11-19T08:37:38.779450+0000 mon.smithi023 (mon.0) 2021-11-19T08:37:39.973 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:39 smithi023 conmon[33490]: 202 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:39.974 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:39 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:38.780029+0000 mgr.smithi023.ehlpcl (mgr.14164) 54 : cephadm [INF] Reconfiguring mgr.smithi023.ehlpcl (unknown last config time)... 2021-11-19T08:37:39.974 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:39 smithi023 conmon[33490]: audit 2021-11-19T08:37:38.780278+0000 mon.smithi023 (mon.0) 203 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi023.ehlpcl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-19T08:37:39.975 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:39 smithi023 conmon[33490]: audit 2021-11-19T08:37:38.781048+0000 mon.smithi023 (mon.0) 204 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-19T08:37:39.975 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:39 smithi023 conmon[33490]: audit 2021-11-19T08:37:38.781609+0000 mon.smithi023 (mon.0) 205 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:40.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:39 smithi134 conmon[36363]: audit 2021-11-19T08:37:38.779450+0000 2021-11-19T08:37:40.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:39 smithi134 conmon[36363]: mon.smithi023 (mon.0) 202 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:40.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:39 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:38.780029+0000 mgr.smithi023.ehlpcl 2021-11-19T08:37:40.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:39 smithi134 conmon[36363]: (mgr.14164) 54 : cephadm [INF] Reconfiguring mgr.smithi023.ehlpcl (unknown last config time)... 2021-11-19T08:37:40.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:39 smithi134 conmon[36363]: audit 2021-11-19T08:37:40.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:39 smithi134 conmon[36363]: 2021-11-19T08:37:38.780278+0000 mon.smithi023 (mon. 2021-11-19T08:37:40.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:39 smithi134 conmon[36363]: 0) 203 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi023.ehlpcl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-19T08:37:40.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:39 smithi134 conmon[36363]: audit 2021-11- 2021-11-19T08:37:40.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:39 smithi134 conmon[36363]: 19T08:37:38.781048+0000 mon.smithi023 (mon 2021-11-19T08:37:40.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:39 smithi134 conmon[36363]: .0) 204 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-19T08:37:40.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:39 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:37:40.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:39 smithi134 conmon[36363]: 11-19T08:37:38.781609+0000 2021-11-19T08:37:40.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:39 smithi134 conmon[36363]: mon.smithi023 (mon.0) 205 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:40.046 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:39 smithi134 conmon[36363]: 2021-11-19T08:37:41.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:40 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:38.782123+0000 mgr.smithi023.ehlpcl (mgr.14164) 55 : cephadm [INF] Reconfiguring daemon mgr.smithi023.ehlpcl on smithi023 2021-11-19T08:37:41.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:40 smithi134 conmon[36363]: cluster 2021-11-19T08:37:41.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:40 smithi134 conmon[36363]: 2021-11-19T08:37:38.797249+0000 mgr.smithi023.ehlpcl (mgr.14164) 56 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:41.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:40 smithi134 conmon[36363]: audit 2021-11-19T08:37:40 2021-11-19T08:37:41.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:40 smithi134 conmon[36363]: .213569+0000 mon.smithi023 (mon.0) 206 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:41.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:40 smithi134 conmon[36363]: cephadm 2021-11-19 2021-11-19T08:37:41.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:40 smithi134 conmon[36363]: T08:37:40.214151+0000 mgr.smithi023.ehlpcl 2021-11-19T08:37:41.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:40 smithi134 conmon[36363]: (mgr.14164) 57 : cephadm [INF] Reconfiguring alertmanager.smithi023 (dependencies changed)... 2021-11-19T08:37:41.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:40 smithi134 conmon[36363]: cephadm 2021-11 2021-11-19T08:37:41.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:40 smithi134 conmon[36363]: -19T08:37:40.216581+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:37:41.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:40 smithi134 conmon[36363]: mgr.14164) 58 : cephadm [INF] Reconfiguring daemon alertmanager.smithi023 on smithi023 2021-11-19T08:37:41.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:40 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:38.782123+0000 mgr.smithi023.ehlpcl (mgr.14164) 55 : cephadm [INF] Reconfiguring daemon mgr.smithi023.ehlpcl on smithi023 2021-11-19T08:37:41.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:40 smithi023 conmon[33490]: cluster 2021-11-19T08:37:38.797249+0000 mgr.smithi023.ehlpcl (mgr.14164) 56 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:41.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:40 smithi023 conmon[33490]: audit 2021-11-19T08:37:40.213569+0000 mon.smithi023 (mon.0) 206 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:41.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:40 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:40.214151+0000 mgr.smithi023.ehlpcl (mgr.14164) 57 : cephadm [INF] Reconfiguring alertmanager.smithi023 (dependencies changed)... 2021-11-19T08:37:41.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:40 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:40.216581+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:37:41.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:40 smithi023 conmon[33490]: ) 58 : cephadm [INF] Reconfiguring daemon alertmanager.smithi023 on smithi023 2021-11-19T08:37:42.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:41 smithi134 conmon[36363]: cluster 2021-11-19T08:37:40.797512+0000 mgr.smithi023.ehlpcl (mgr.14164) 59 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:42.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:41 smithi134 conmon[36363]: audit 2021-11-19T08:37:41.312946+0000 mon.smithi023 (mon.0) 207 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-11-19T08:37:42.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:41 smithi134 conmon[36363]: audit 2021-11-19T08 2021-11-19T08:37:42.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:41 smithi134 conmon[36363]: :37:41.313661+0000 mon.smithi023 (mon.0) 208 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:42.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:41 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:42.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:41 smithi134 conmon[36363]: 2021-11-19T08:37: 2021-11-19T08:37:42.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:41 smithi134 conmon[36363]: 41.314241+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:37:42.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:41 smithi134 conmon[36363]: ) 60 : cephadm [INF] Deploying daemon osd.0 on smithi023 2021-11-19T08:37:42.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:41 smithi023 conmon[33490]: cluster 2021-11-19T08:37:40.797512+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:37:42.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:41 smithi023 conmon[33490]: .14164) 59 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:42.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:41 smithi023 conmon[33490]: audit 2021-11-19T08:37:41.312946+0000 mon.smithi023 (mon.0) 207 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-11-19T08:37:42.145 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:41 smithi023 conmon[33490]: audit 2021-11-19T08:37:41.313661+0000 mon.smithi023 (mon.0) 208 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:42.145 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:41 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:41.314241+0000 mgr.smithi023.ehlpcl (mgr.14164) 60 : cephadm [INF] Deploying daemon osd.0 on smithi023 2021-11-19T08:37:43.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:43 smithi023 conmon[33490]: cluster 2021-11-19T08:37:42.797737+0000 2021-11-19T08:37:43.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:43 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 61 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:44.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:43 smithi134 conmon[36363]: cluster 2021-11-19T08:37:42.797737+0000 mgr.smithi023.ehlpcl (mgr.14164) 61 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:45.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:45 smithi023 conmon[33490]: audit 2021-11-19T08:37:44.328500+0000 mon.smithi023 2021-11-19T08:37:45.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:45 smithi023 conmon[33490]: (mon.0) 209 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:45.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:45 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:44.329003+0000 mgr.smithi023.ehlpcl (mgr.14164) 62 : cephadm [INF] Reconfiguring crash.smithi023 (monmap changed)... 2021-11-19T08:37:45.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:45 smithi023 conmon[33490]: audit 2021-11-19T08:37:44.329251+0000 mon.smithi023 (mon.0) 210 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi023", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-19T08:37:45.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:45 smithi023 conmon[33490]: audit 2021-11-19T08:37:44.329934+0000 mon.smithi023 (mon.0) 211 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:45.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:45 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:44.330445+0000 mgr.smithi023.ehlpcl (mgr.14164) 63 : cephadm [INF] Reconfiguring daemon crash.smithi023 on smithi023 2021-11-19T08:37:45.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:45 smithi134 conmon[36363]: audit 2021-11-19T08:37:44.328500+0000 mon.smithi023 (mon.0) 209 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:45.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:45 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:45.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:45 smithi134 conmon[36363]: 2021-11-19T08:37:44.329003+0000 mgr.smithi023.ehlpcl (mgr.14164) 62 : cephadm [INF] Reconfiguring crash.smithi023 (monmap changed)... 2021-11-19T08:37:45.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:45 smithi134 conmon[36363]: audit 2021-11-19T08:37:44.329251 2021-11-19T08:37:45.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:45 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 210 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi023", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-19T08:37:45.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:45 smithi134 conmon[36363]: audit 2021-11- 2021-11-19T08:37:45.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:45 smithi134 conmon[36363]: 19T08:37:44.329934+0000 2021-11-19T08:37:45.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:45 smithi134 conmon[36363]: mon.smithi023 (mon.0) 211 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:45.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:45 smithi134 conmon[36363]: 2021-11-19T08:37:45.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:45 smithi134 conmon[36363]: cephadm 2021-11-19T 2021-11-19T08:37:45.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:45 smithi134 conmon[36363]: 08:37:44.330445+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:37:45.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:45 smithi134 conmon[36363]: .14164) 63 : cephadm [INF] Reconfiguring daemon crash.smithi023 on smithi023 2021-11-19T08:37:46.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:46 smithi023 conmon[33490]: cluster 2021-11-19T08:37:44.797967+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:37:46.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:46 smithi023 conmon[33490]: 64 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:46.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:46 smithi134 conmon[36363]: cluster 2021-11-19T08:37:44.797967+0000 mgr.smithi023.ehlpcl (mgr.14164) 64 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:47.321 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 0 on host 'smithi023' 2021-11-19T08:37:47.322 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:47.333+0000 7f3911ffb700 1 -- 172.21.15.23:0/870336013 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f391c101140 con 0x7f390805c2b0 2021-11-19T08:37:47.323 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:47.336+0000 7f392389b700 1 -- 172.21.15.23:0/870336013 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f390805c2b0 msgr2=0x7f390805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:47.323 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:47.336+0000 7f392389b700 1 --2- 172.21.15.23:0/870336013 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f390805c2b0 0x7f390805e770 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3918006fd0 tx=0x7f3918008040).stop 2021-11-19T08:37:47.323 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:47.336+0000 7f392389b700 1 -- 172.21.15.23:0/870336013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f391c0fbc50 msgr2=0x7f391c1096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:47.323 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:47.336+0000 7f392389b700 1 --2- 172.21.15.23:0/870336013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f391c0fbc50 0x7f391c1096d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f390c004e20 tx=0x7f390c00b040).stop 2021-11-19T08:37:47.324 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:47.336+0000 7f392389b700 1 -- 172.21.15.23:0/870336013 shutdown_connections 2021-11-19T08:37:47.324 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:47.336+0000 7f392389b700 1 --2- 172.21.15.23:0/870336013 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f391c060390 0x7f391c1091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:47.324 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:47.336+0000 7f392389b700 1 --2- 172.21.15.23:0/870336013 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f390805c2b0 0x7f390805e770 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:47.325 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:47.336+0000 7f392389b700 1 --2- 172.21.15.23:0/870336013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f391c0fbc50 0x7f391c1096d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:47.325 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:47.336+0000 7f392389b700 1 -- 172.21.15.23:0/870336013 >> 172.21.15.23:0/870336013 conn(0x7f391c0f0de0 msgr2=0x7f391c0f31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:37:47.325 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:47.337+0000 7f392389b700 1 -- 172.21.15.23:0/870336013 shutdown_connections 2021-11-19T08:37:47.325 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:47.337+0000 7f392389b700 1 -- 172.21.15.23:0/870336013 wait complete. 2021-11-19T08:37:47.723 DEBUG:teuthology.orchestra.run.smithi023:osd.0> sudo journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.0.service 2021-11-19T08:37:47.726 INFO:tasks.cephadm:Deploying osd.1 on smithi023 with /dev/vg_nvme/lv_3... 2021-11-19T08:37:47.727 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-11-19T08:37:47.756 INFO:journalctl@ceph.osd.0.smithi023.stdout:-- Logs begin at Fri 2021-11-19 08:24:48 UTC. -- 2021-11-19T08:37:47.913 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 19 08:37:47 smithi023 conmon[50499]: debug 2021-11-19T08:37:47.778+0000 7ff6543f0080 -1 Falling back to public interface 2021-11-19T08:37:48.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:48 smithi023 conmon[33490]: cluster 2021-11-19T08:37:46.798253+0000 2021-11-19T08:37:48.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:48 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 65 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:48.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:48 smithi023 conmon[33490]: audit 2021-11-19T08:37:47.325480+0000 mon.smithi023 (mon.0) 212 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:48.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:48 smithi023 conmon[33490]: audit 2021-11-19T08:37:47.333550+0000 mon.smithi023 (mon.0) 213 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:48.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:48 smithi023 conmon[33490]: audit 2021-11-19T08:37:48.265277+0000 mon.smithi023 (mon.0) 214 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:48.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:48 smithi134 conmon[36363]: cluster 2021-11-19T08:37:46.798253+0000 mgr.smithi023.ehlpcl (mgr.14164) 65 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:48.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:48 smithi134 conmon[36363]: audit 2021-11-19T08:37:47.325480+0000 mon.smithi023 (mon.0) 212 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:48.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:48 smithi134 conmon[36363]: audit 2021-11-19T08 2021-11-19T08:37:48.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:48 smithi134 conmon[36363]: :37:47.333550+0000 mon.smithi023 (mon.0) 213 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:48.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:48 smithi134 conmon[36363]: audit 2021-11 2021-11-19T08:37:48.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:48 smithi134 conmon[36363]: -19T08:37:48.265277+0000 mon.smithi023 2021-11-19T08:37:48.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:48 smithi134 conmon[36363]: (mon.0) 214 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:49.629 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:49 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:48.265866+0000 mgr.smithi023.ehlpcl (mgr.14164) 66 : cephadm [INF] Reconfiguring grafana.smithi023 (dependencies changed)... 2021-11-19T08:37:49.630 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:49 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:48.270288+0000 mgr.smithi023.ehlpcl (mgr.14164) 67 : cephadm [INF] Reconfiguring daemon grafana.smithi023 on smithi023 2021-11-19T08:37:49.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:49 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:48.265866+0000 mgr.smithi023.ehlpcl (mgr.14164) 66 : cephadm [INF] Reconfiguring grafana.smithi023 (dependencies changed)... 2021-11-19T08:37:49.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:49 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:48.270288+0000 mgr.smithi023.ehlpcl (mgr.14164) 67 : cephadm 2021-11-19T08:37:49.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:49 smithi134 conmon[36363]: [INF] Reconfiguring daemon grafana.smithi023 on smithi023 2021-11-19T08:37:49.810 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-11-19T08:37:49.810 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-11-19T08:37:49.810 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2021-11-19T08:37:49.811 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2021-11-19T08:37:49.811 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116153 s, 903 MB/s 2021-11-19T08:37:49.811 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2021-11-19T08:37:50.443 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi023:vg_nvme/lv_3 2021-11-19T08:37:50.583 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:50 smithi023 conmon[33490]: cluster 2021-11-19T08:37:48.798726+0000 mgr.smithi023.ehlpcl (mgr.14164) 68 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:50.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:50 smithi134 conmon[36363]: cluster 2021-11-19T08:37:48.798726+0000 mgr.smithi023.ehlpcl (mgr.14164) 68 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:51.521 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:51 smithi023 conmon[33490]: audit 2021-11-19T08:37:51.288960+0000 mon.smithi023 (mon.0) 215 : audit [INF] from='osd.0 [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-11-19T08:37:51.522 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 19 08:37:51 smithi023 conmon[50499]: debug 2021-11-19T08:37:51.285+0000 7ff6543f0080 -1 osd.0 0 log_to_monitors {default=true} 2021-11-19T08:37:51.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:51 smithi134 conmon[36363]: audit 2021-11-19T08:37:51.288960+0000 mon.smithi023 (mon.0) 215 : audit [INF] from='osd.0 [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-11-19T08:37:51.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:51 smithi134 conmon[36363]: 2021-11-19T08:37:52.569 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.581+0000 7fae372f3700 1 -- 172.21.15.23:0/3207316744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae300f55c0 msgr2=0x7fae300f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:52.569 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.581+0000 7fae372f3700 1 --2- 172.21.15.23:0/3207316744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae300f55c0 0x7fae300f59e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fae20004660 tx=0x7fae20009b30).stop 2021-11-19T08:37:52.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.581+0000 7fae372f3700 1 -- 172.21.15.23:0/3207316744 shutdown_connections 2021-11-19T08:37:52.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.581+0000 7fae372f3700 1 --2- 172.21.15.23:0/3207316744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae300f6860 0x7fae300f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:52.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.581+0000 7fae372f3700 1 --2- 172.21.15.23:0/3207316744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae300f55c0 0x7fae300f59e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:52.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.581+0000 7fae372f3700 1 -- 172.21.15.23:0/3207316744 >> 172.21.15.23:0/3207316744 conn(0x7fae300f0d40 msgr2=0x7fae300f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:37:52.571 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.581+0000 7fae372f3700 1 -- 172.21.15.23:0/3207316744 shutdown_connections 2021-11-19T08:37:52.571 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.581+0000 7fae372f3700 1 -- 172.21.15.23:0/3207316744 wait complete. 2021-11-19T08:37:52.571 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.582+0000 7fae372f3700 1 Processor -- start 2021-11-19T08:37:52.571 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.582+0000 7fae372f3700 1 -- start start 2021-11-19T08:37:52.572 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.582+0000 7fae372f3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae300f55c0 0x7fae3010c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:52.572 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.582+0000 7fae372f3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae300f6860 0x7fae3010e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:52.572 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.582+0000 7fae372f3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae300602b0 con 0x7fae300f6860 2021-11-19T08:37:52.572 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.582+0000 7fae372f3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae300603f0 con 0x7fae300f55c0 2021-11-19T08:37:52.573 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.582+0000 7fae3488e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae300f6860 0x7fae3010e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:52.573 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.582+0000 7fae3488e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae300f6860 0x7fae3010e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46432/0 (socket says 172.21.15.23:46432) 2021-11-19T08:37:52.573 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.582+0000 7fae3488e700 1 -- 172.21.15.23:0/853764501 learned_addr learned my addr 172.21.15.23:0/853764501 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:37:52.573 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.582+0000 7fae3508f700 1 --2- 172.21.15.23:0/853764501 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae300f55c0 0x7fae3010c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.582+0000 7fae3488e700 1 -- 172.21.15.23:0/853764501 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae300f55c0 msgr2=0x7fae3010c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:37:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.582+0000 7fae3488e700 1 --2- 172.21.15.23:0/853764501 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae300f55c0 0x7fae3010c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:37:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.582+0000 7fae3488e700 1 -- 172.21.15.23:0/853764501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae20005040 con 0x7fae300f6860 2021-11-19T08:37:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.583+0000 7fae3488e700 1 --2- 172.21.15.23:0/853764501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae300f6860 0x7fae3010e5b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fae2c00c790 tx=0x7fae2c007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:37:52.575 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.583+0000 7fae25ffb700 1 -- 172.21.15.23:0/853764501 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae2c00e680 con 0x7fae300f6860 2021-11-19T08:37:52.575 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.583+0000 7fae372f3700 1 -- 172.21.15.23:0/853764501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3010eaa0 con 0x7fae300f6860 2021-11-19T08:37:52.575 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.583+0000 7fae25ffb700 1 -- 172.21.15.23:0/853764501 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae2c00ec90 con 0x7fae300f6860 2021-11-19T08:37:52.575 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.583+0000 7fae25ffb700 1 -- 172.21.15.23:0/853764501 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae2c013d60 con 0x7fae300f6860 2021-11-19T08:37:52.576 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.583+0000 7fae372f3700 1 -- 172.21.15.23:0/853764501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3010ef20 con 0x7fae300f6860 2021-11-19T08:37:52.577 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.584+0000 7fae372f3700 1 -- 172.21.15.23:0/853764501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae30045ad0 con 0x7fae300f6860 2021-11-19T08:37:52.577 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.584+0000 7fae25ffb700 1 -- 172.21.15.23:0/853764501 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fae2c01a060 con 0x7fae300f6860 2021-11-19T08:37:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.585+0000 7fae25ffb700 1 --2- 172.21.15.23:0/853764501 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fae1c05c2b0 0x7fae1c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:37:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.585+0000 7fae25ffb700 1 -- 172.21.15.23:0/853764501 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1602+0+0 (secure 0 0 0) 0x7fae2c075220 con 0x7fae300f6860 2021-11-19T08:37:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.585+0000 7fae3508f700 1 --2- 172.21.15.23:0/853764501 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fae1c05c2b0 0x7fae1c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:37:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.585+0000 7fae3508f700 1 --2- 172.21.15.23:0/853764501 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fae1c05c2b0 0x7fae1c05e770 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fae20009d60 tx=0x7fae20009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:37:52.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.589+0000 7fae25ffb700 1 -- 172.21.15.23:0/853764501 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae2c048d40 con 0x7fae300f6860 2021-11-19T08:37:52.642 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 19 08:37:52 smithi023 conmon[50499]: debug 2021-11-19T08:37:52.349+0000 7ff64a961700 -1 osd.0 0 waiting for initial osdmap 2021-11-19T08:37:52.642 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 19 08:37:52 smithi023 conmon[50499]: debug 2021-11-19T08:37:52.353+0000 7ff6468dc700 -1 osd.0 8 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2021-11-19T08:37:52.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:52 smithi023 conmon[33490]: cluster 2021-11-19T08:37:50.798966+0000 mgr.smithi023.ehlpcl (mgr.14164) 69 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:52.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:52 smithi023 conmon[33490]: audit 2021-11-19T08:37:51.343284+0000 mon.smithi023 (mon.0) 216 : audit [INF] from='osd.0 [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-11-19T08:37:52.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:52 smithi023 conmon[33490]: cluster 2021-11-19T08:37:51.343336+0000 mon.smithi023 (mon.0) 217 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-11-19T08:37:52.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:52 smithi023 conmon[33490]: audit 2021-11-19T08:37:51.343448+0000 mon.smithi023 (mon.0) 218 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-19T08:37:52.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:52 smithi023 conmon[33490]: audit 2021-11-19T08:37:51.343833+0000 mon.smithi023 (mon.0) 219 : audit [INF] from='osd.0 [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2021-11-19T08:37:52.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:52 smithi023 conmon[33490]: audit 2021-11-19T08:37:52.248509+0000 mon.smithi023 (mon.0) 220 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:52.761 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:37:52.776+0000 7fae372f3700 1 -- 172.21.15.23:0/853764501 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fae300fdf00 con 0x7fae1c05c2b0 2021-11-19T08:37:52.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:52 smithi134 conmon[36363]: cluster 2021-11-19T08:37:50.798966+0000 mgr.smithi023.ehlpcl (mgr.14164) 69 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:52.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:52 smithi134 conmon[36363]: audit 2021-11-19 2021-11-19T08:37:52.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:52 smithi134 conmon[36363]: T08:37:51.343284+0000 mon.smithi023 (mon.0) 216 : audit [INF] from='osd.0 [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-11-19T08:37:52.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:52 smithi134 conmon[36363]: cluster 2021-11-19T08:37:51.343336+0000 mon.smithi023 (mon.0) 217 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-11-19T08:37:52.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:52 smithi134 conmon[36363]: audit 2021-11-19T08:37:51.343448+0000 2021-11-19T08:37:52.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:52 smithi134 conmon[36363]: mon.smithi023 (mon.0) 218 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-19T08:37:52.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:52 smithi134 conmon[36363]: 2021-11-19T08:37:52.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:52 smithi134 conmon[36363]: audit 2021-11- 2021-11-19T08:37:52.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:52 smithi134 conmon[36363]: 19T08:37:51.343833+0000 mon.smithi023 (mon.0) 219 2021-11-19T08:37:52.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:52 smithi134 conmon[36363]: : audit [INF] from='osd.0 [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2021-11-19T08:37:52.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:52 smithi134 conmon[36363]: audit 2021-11-19T08:37:52.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:52 smithi134 conmon[36363]: 2021-11-19T08:37:52.248509+0000 mon.smithi023 (mon 2021-11-19T08:37:52.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:52 smithi134 conmon[36363]: .0) 220 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:53.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:53 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:52.249201+0000 mgr.smithi023.ehlpcl (mgr.14164) 70 : cephadm [INF] Reconfiguring prometheus.smithi023 (dependencies changed)... 2021-11-19T08:37:53.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:53 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:52.251725+0000 mgr.smithi023.ehlpcl (mgr.14164) 71 : cephadm [INF] Reconfiguring daemon prometheus.smithi023 on smithi023 2021-11-19T08:37:53.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:53 smithi023 conmon[33490]: audit 2021-11-19T08:37:52.342924+0000 mon.smithi023 (mon.0) 221 : audit [INF] from='osd.0 [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2021-11-19T08:37:53.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:53 smithi023 conmon[33490]: cluster 2021-11-19T08:37:52.342986+0000 mon.smithi023 (mon.0) 222 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2021-11-19T08:37:53.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:53 smithi023 conmon[33490]: audit 2021-11-19T08:37:52.343438+0000 mon.smithi023 (mon.0) 223 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-19T08:37:53.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:53 smithi023 conmon[33490]: audit 2021-11-19T08:37:52.350617+0000 mon.smithi023 (mon.0) 224 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-19T08:37:53.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:53 smithi023 conmon[33490]: audit 2021-11-19T08:37:52.777738+0000 mgr.smithi023.ehlpcl (mgr.14164) 72 : audit [DBG] from='client.14230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:37:53.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:53 smithi023 conmon[33490]: audit 2021-11-19T08:37:52.778497+0000 mon.smithi023 (mon.0) 225 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-19T08:37:53.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:53 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:52.779683+0000 mgr.smithi023.ehlpcl (mgr.14164) 73 : cephadm [INF] Applying service osd.None on host smithi023... 2021-11-19T08:37:53.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:53 smithi023 conmon[33490]: audit 2021-11-19T08:37:52.779959+0000 mon.smithi023 ( 2021-11-19T08:37:53.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:53 smithi023 conmon[33490]: mon.0) 226 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-19T08:37:53.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:53 smithi023 conmon[33490]: audit 2021-11-19T08:37:52.780597+0000 mon.smithi023 (mon.0) 227 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:53.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:52.249201+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:37:53.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: .14164) 70 : cephadm [INF] Reconfiguring prometheus.smithi023 (dependencies changed)... 2021-11-19T08:37:53.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:52.251725+0000 mgr.smithi023.ehlpcl (mgr.14164) 71 : cephadm [INF] Reconfiguring daemon prometheus.smithi023 on smithi023 2021-11-19T08:37:53.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: audit 2021-11-19T08:37:52.342924+0000 2021-11-19T08:37:53.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: mon.smithi023 (mon.0) 221 : audit [INF] from='osd.0 [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2021-11-19T08:37:53.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: cluster 2021- 2021-11-19T08:37:53.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: 11-19T08:37:52.342986+0000 2021-11-19T08:37:53.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: mon.smithi023 (mon.0) 222 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2021-11-19T08:37:53.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: 2021-11-19T08:37:53.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: audit 2021-11-19T08:37:53.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: 2021-11-19T08:37:52.343438+0000 mon.smithi023 ( 2021-11-19T08:37:53.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: mon.0) 223 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-19T08:37:53.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: audit 2021 2021-11-19T08:37:53.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: -11-19T08:37:52.350617+0000 2021-11-19T08:37:53.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: mon.smithi023 (mon.0) 224 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-19T08:37:53.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: 2021-11-19T08:37:53.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: audit 2021-11-19T08:37:52. 2021-11-19T08:37:53.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: 777738+0000 mgr.smithi023.ehlpcl (mgr.14164) 72 : audit [DBG] 2021-11-19T08:37:53.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: from='client.14230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:37:53.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: audit 2021-11-19T08:37:52. 2021-11-19T08:37:53.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: 778497+0000 mon.smithi023 (mon.0) 225 : audit 2021-11-19T08:37:53.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-19T08:37:53.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: cephadm 2021-11-19T08: 2021-11-19T08:37:53.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: 37:52.779683+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:37:53.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: 73 : cephadm [INF] Applying service osd.None on host smithi023... 2021-11-19T08:37:53.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: audit 2021-11 2021-11-19T08:37:53.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: -19T08:37:52.779959+0000 mon.smithi023 2021-11-19T08:37:53.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: (mon.0) 226 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-19T08:37:53.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: 2021-11-19T08:37:53.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: audit 2021-11-19T08:37: 2021-11-19T08:37:53.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: 52.780597+0000 mon.smithi023 (mon.0) 227 : 2021-11-19T08:37:53.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:53 smithi134 conmon[36363]: audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:54.537 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: cluster 2021-11-19T08:37:52.252703+0000 2021-11-19T08:37:54.537 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-19T08:37:54.538 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: cluster 2021-11-19T08:37: 2021-11-19T08:37:54.538 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: 52.252793+0000 osd.0 (osd.0) 2 : cluster [DBG] 2021-11-19T08:37:54.538 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: purged_snaps scrub ok 2021-11-19T08:37:54.539 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: cluster 2021- 2021-11-19T08:37:54.539 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: 11-19T08:37:52 2021-11-19T08:37:54.539 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: .799189+0000 mgr.smithi023.ehlpcl (mgr.14164) 74 2021-11-19T08:37:54.539 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:54.539 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: cluster 2021-11-19T08:37:54.540 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: 2021-11-19T08:37 2021-11-19T08:37:54.540 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: :53.352431+0000 mon.smithi023 (mon.0 2021-11-19T08:37:54.540 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: ) 228 : cluster [INF] osd.0 [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305] boot 2021-11-19T08:37:54.540 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: cluster 2021 2021-11-19T08:37:54.541 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: -11-19T08:37: 2021-11-19T08:37:54.541 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: 53.352461+0000 mon.smithi023 (mon 2021-11-19T08:37:54.541 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: .0) 229 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-11-19T08:37:54.541 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: 2021-11-19T08:37:54.541 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:37:54.542 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: T08:37:53.352698+0000 2021-11-19T08:37:54.542 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: mon.smithi023 (mon.0) 230 : audit [DBG] 2021-11-19T08:37:54.542 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:54 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-19T08:37:54.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:54 smithi134 conmon[36363]: cluster 2021-11-19T08:37:52.252703+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-19T08:37:54.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:54 smithi134 conmon[36363]: cluster 2021-11- 2021-11-19T08:37:54.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:54 smithi134 conmon[36363]: 19T08:37:52.252793+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-19T08:37:54.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:54 smithi134 conmon[36363]: cluster 2021-11-19T08:37:52.799189+0000 mgr.smithi023.ehlpcl (mgr.14164) 74 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-11-19T08:37:54.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:54 smithi134 conmon[36363]: cluster 2021-11-19T08:37:53.352431+0000 mon.smithi023 (mon.0) 228 : cluster [INF] osd.0 [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305] boot 2021-11-19T08:37:54.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:54 smithi134 conmon[36363]: cluster 2021-11-19T08:37:53.352461+0000 mon.smithi023 (mon.0) 229 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-11-19T08:37:54.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:54 smithi134 conmon[36363]: audit 2021-11-19T08:37:53. 2021-11-19T08:37:54.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:54 smithi134 conmon[36363]: 352698+0000 mon.smithi023 (mon.0) 230 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-11-19T08:37:56.247 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:56 smithi023 conmon[33490]: cluster 2021-11-19T08:37:54.799448+0000 mgr.smithi023.ehlpcl (mgr.14164) 75 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:37:56.247 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:56 smithi023 conmon[33490]: cluster 2021-11-19T08:37:55.141718+0000 mon.smithi023 (mon.0) 231 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2021-11-19T08:37:56.248 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:56 smithi023 conmon[33490]: audit 2021-11-19T08:37:56.061792+0000 mon.smithi023 (mon.0) 232 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:56.248 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:56 smithi023 conmon[33490]: audit 2021-11-19T08:37:56.062545+0000 mon.smithi023 (mon.0) 233 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-19T08:37:56.248 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:56 smithi023 conmon[33490]: audit 2021-11-19T08:37:56.063221+0000 mon.smithi023 (mon.0) 234 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:56.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:56 smithi134 conmon[36363]: cluster 2021-11-19T08:37:54.799448+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:37:56.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:56 smithi134 conmon[36363]: 75 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:37:56.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:56 smithi134 conmon[36363]: cluster 2021-11-19T08:37:55.141718+0000 mon.smithi023 (mon.0) 231 : cluster [DBG] osdmap e10: 1 total, 1 up, 1 in 2021-11-19T08:37:56.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:56 smithi134 conmon[36363]: audit 2021-11-19T08:37:56 2021-11-19T08:37:56.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:56 smithi134 conmon[36363]: .061792+0000 mon.smithi023 (mon.0) 232 2021-11-19T08:37:56.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:56 smithi134 conmon[36363]: : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:56.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:56 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:37:56.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:56 smithi134 conmon[36363]: 11-19T08:37:56.062545 2021-11-19T08:37:56.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:56 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 233 2021-11-19T08:37:56.488 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:56 smithi134 conmon[36363]: : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2021-11-19T08:37:56.488 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:56 smithi134 conmon[36363]: audit 2021 2021-11-19T08:37:56.488 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:56 smithi134 conmon[36363]: -11-19T08:37:56. 2021-11-19T08:37:56.488 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:56 smithi134 conmon[36363]: 063221+0000 mon.smithi023 (mon.0) 2021-11-19T08:37:56.488 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:56 smithi134 conmon[36363]: 234 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:57.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:57 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:56. 2021-11-19T08:37:57.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:57 smithi023 conmon[33490]: 062321+0000 mgr.smithi023.ehlpcl (mgr.14164) 76 : cephadm [INF] Reconfiguring crash.smithi134 (monmap changed)... 2021-11-19T08:37:57.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:57 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:56.063716+0000 mgr.smithi023.ehlpcl (mgr.14164) 77 : cephadm [INF] Reconfiguring daemon crash.smithi134 on smithi134 2021-11-19T08:37:57.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:57 smithi023 conmon[33490]: audit 2021-11-19T08:37:57.040067+0000 mon.smithi023 (mon.0) 235 : audit [INF] from='client.? 172.21.15.23:0/1330484598' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b7aea87-24b7-43e6-9ade-8fef0e9e6dd2"}]: dispatch 2021-11-19T08:37:57.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:57 smithi023 conmon[33490]: audit 2021-11-19T08:37:57.045587+0000 mon.smithi023 (mon.0) 236 : audit [INF] from='client.? 172.21.15.23:0/1330484598' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b7aea87-24b7-43e6-9ade-8fef0e9e6dd2"}]': finished 2021-11-19T08:37:57.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:57 smithi023 conmon[33490]: cluster 2021-11-19T08:37:57.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:57 smithi023 conmon[33490]: 2021-11-19T08:37:57.045630+0000 mon.smithi023 (mon.0) 237 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-11-19T08:37:57.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:57 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:37:57.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:57 smithi023 conmon[33490]: 11-19T08:37:57.045776+0000 mon.smithi023 2021-11-19T08:37:57.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:57 smithi023 conmon[33490]: (mon.0) 238 : audit [DBG] 2021-11-19T08:37:57.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:57 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-19T08:37:57.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:57 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:56.062321+0000 mgr.smithi023.ehlpcl (mgr.14164) 76 : cephadm [INF] Reconfiguring crash.smithi134 (monmap changed)... 2021-11-19T08:37:57.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:57 smithi134 conmon[36363]: cephadm 2021-11-19T 2021-11-19T08:37:57.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:57 smithi134 conmon[36363]: 08:37:56.063716+0000 mgr.smithi023.ehlpcl (mgr.14164) 77 : cephadm [INF] Reconfiguring daemon crash.smithi134 on smithi134 2021-11-19T08:37:57.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:57 smithi134 conmon[36363]: audit 2021-11-19T08:37:57.040067+0000 mon.smithi023 (mon.0) 235 : audit [INF] from='client.? 172.21.15.23:0/1330484598' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b7aea87-24b7-43e6-9ade-8fef0e9e6dd2"}]: dispatch 2021-11-19T08:37:57.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:57 smithi134 conmon[36363]: audit 2021-11-19T08:37:57.045587+0000 mon.smithi023 (mon.0) 236 : audit [INF] from='client.? 172.21.15.23:0/1330484598' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b7aea87-24b7-43e6-9ade-8fef0e9e6dd2"}]': finished 2021-11-19T08:37:57.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:57 smithi134 conmon[36363]: cluster 2021-11-19T08:37:57.045630+0000 mon.smithi023 (mon.0) 237 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-11-19T08:37:57.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:57 smithi134 conmon[36363]: audit 2021-11-19T08:37:57.045776+0000 mon.smithi023 (mon.0) 238 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-19T08:37:58.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: cluster 2021-11-19T08:37:56.799729+0000 mgr.smithi023.ehlpcl (mgr.14164) 78 : cluster 2021-11-19T08:37:58.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: [DBG] pgmap v48: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:37:58.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: audit 2021-11-19T08:37:57.150107+0000 mon.smithi023 (mon.0) 239 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:58.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:57.150688+0000 mgr.smithi023.ehlpcl (mgr.14164) 79 : cephadm [INF] Reconfiguring mgr.smithi134.vitoqy (monmap changed)... 2021-11-19T08:37:58.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: audit 2021-11-19T08:37:57.150967+0000 mon.smithi023 (mon.0) 240 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.vitoqy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-19T08:37:58.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: audit 2021-11-19T08:37:57.151720+0000 mon.smithi023 ( 2021-11-19T08:37:58.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: mon.0) 241 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-19T08:37:58.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:37:58.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: T08:37:57.152361+0000 2021-11-19T08:37:58.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: mon.smithi023 (mon.0) 242 : audit 2021-11-19T08:37:58.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:58.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: cephadm 2021-11 2021-11-19T08:37:58.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: -19T08:37:57.152920+0000 2021-11-19T08:37:58.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 80 : cephadm 2021-11-19T08:37:58.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: [INF] Reconfiguring daemon mgr.smithi134.vitoqy on smithi134 2021-11-19T08:37:58.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: audit 2021 2021-11-19T08:37:58.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: -11-19T08:37:57 2021-11-19T08:37:58.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: .473945+0000 mon.smithi023 (mon. 2021-11-19T08:37:58.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: 0) 243 : audit [DBG] from='client.? 172.21.15.23:0/972695369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-19T08:37:58.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:58 smithi023 conmon[33490]: 2021-11-19T08:37:58.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:58 smithi134 conmon[36363]: cluster 2021-11-19T08:37:56.799729+0000 mgr.smithi023.ehlpcl (mgr.14164) 78 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:37:58.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:58 smithi134 conmon[36363]: audit 2021-11-19T08:37:57.150107+0000 mon.smithi023 (mon.0) 239 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:58.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:58 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:57.150688+0000 mgr.smithi023.ehlpcl (mgr.14164) 79 : cephadm [INF] Reconfiguring mgr.smithi134.vitoqy (monmap changed)... 2021-11-19T08:37:58.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:58 smithi134 conmon[36363]: audit 2021-11-19T08:37:57.150967+0000 mon.smithi023 (mon.0) 240 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.vitoqy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-11-19T08:37:58.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:58 smithi134 conmon[36363]: audit 2021-11-19T08:37:57.151720+0000 mon.smithi023 (mon.0) 241 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mgr services"}]: dispatch 2021-11-19T08:37:58.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:58 smithi134 conmon[36363]: audit 2021-11-19T08:37:57.152361+0000 mon.smithi023 (mon.0) 242 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:58.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:58 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:57.152920+0000 mgr.smithi023.ehlpcl (mgr.14164) 80 : cephadm [INF] Reconfiguring daemon mgr.smithi134.vitoqy on smithi134 2021-11-19T08:37:58.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:58 smithi134 conmon[36363]: audit 2021-11-19T08:37:57.473945+0000 mon.smithi023 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.23:0/972695369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-19T08:37:59.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: audit 2021-11-19T08:37:58.555686+0000 mon.smithi023 (mon.0) 244 : 2021-11-19T08:37:59.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:37:59.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: cephadm 2021-11-19T08:37:58.556339+0000 mgr.smithi023.ehlpcl (mgr.14164) 81 : cephadm [INF] Reconfiguring mon.smithi134 (monmap changed)... 2021-11-19T08:37:59.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:37:59.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: -19T08:37:58.556689+0000 mon.smithi023 (mon.0) 245 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-19T08:37:59.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:37:59.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: 11-19T08:37:58.557673+0000 2021-11-19T08:37:59.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: mon.smithi023 (mon.0) 246 : audit 2021-11-19T08:37:59.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-19T08:37:59.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:37:59.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: 11-19T08: 2021-11-19T08:37:59.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: 37:58.558640+0000 mon.smithi023 ( 2021-11-19T08:37:59.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: mon.0) 247 : audit [DBG] 2021-11-19T08:37:59.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:37:59.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: cephadm 2021 2021-11-19T08:37:59.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: -11-19T08:37:58. 2021-11-19T08:37:59.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: 559415+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:37:59.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:37:59 smithi023 conmon[33490]: ) 82 : cephadm [INF] Reconfiguring daemon mon.smithi134 on smithi134 2021-11-19T08:38:00.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:59 smithi134 conmon[36363]: audit 2021-11-19T08:37:58.555686+0000 mon.smithi023 (mon.0) 244 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:00.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:59 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:58.556339+0000 mgr.smithi023.ehlpcl (mgr.14164) 81 : cephadm [INF] Reconfiguring mon.smithi134 (monmap changed)... 2021-11-19T08:38:00.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:59 smithi134 conmon[36363]: audit 2021-11-19T08:37:58.556689+0000 mon.smithi023 (mon.0) 245 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-11-19T08:38:00.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:59 smithi134 conmon[36363]: audit 2021-11-19T08:37:58.557673+0000 mon.smithi023 (mon.0) 246 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2021-11-19T08:38:00.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:59 smithi134 conmon[36363]: audit 2021-11-19T08:37:58.558640+0000 mon.smithi023 (mon.0) 247 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:00.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:37:59 smithi134 conmon[36363]: cephadm 2021-11-19T08:37:58.559415+0000 mgr.smithi023.ehlpcl (mgr.14164) 82 : cephadm [INF] Reconfiguring daemon mon.smithi134 on smithi134 2021-11-19T08:38:00.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:00 smithi023 conmon[33490]: cluster 2021-11-19T08:37:58.800031+0000 mgr.smithi023.ehlpcl (mgr.14164) 83 2021-11-19T08:38:00.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:00 smithi023 conmon[33490]: : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:00.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:00 smithi023 conmon[33490]: audit 2021-11-19T08:37:59. 2021-11-19T08:38:00.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:00 smithi023 conmon[33490]: 625347+0000 mon.smithi023 (mon.0) 248 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:00.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:00 smithi023 conmon[33490]: audit 2021-11-19T08:37: 2021-11-19T08:38:00.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:00 smithi023 conmon[33490]: 59.628302+0000 mon.smithi023 (mon.0) 249 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-19T08:38:00.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:00 smithi023 conmon[33490]: audit 2021-11-19T08:37:59. 2021-11-19T08:38:00.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:00 smithi023 conmon[33490]: 628698+0000 mgr.smithi023.ehlpcl (mgr.14164) 84 : audit 2021-11-19T08:38:00.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:00 smithi023 conmon[33490]: [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-19T08:38:01.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:00 smithi134 conmon[36363]: cluster 2021-11-19T08:37:58.800031+0000 mgr.smithi023.ehlpcl (mgr.14164) 83 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:01.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:00 smithi134 conmon[36363]: audit 2021-11-19T08:37:59.625347+0000 mon.smithi023 (mon.0) 248 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:01.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:00 smithi134 conmon[36363]: audit 2021-11-19T08:37:59.628302+0000 mon.smithi023 (mon. 2021-11-19T08:38:01.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:00 smithi134 conmon[36363]: 0) 249 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-19T08:38:01.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:00 smithi134 conmon[36363]: audit 2021-11-19T08:38:01.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:00 smithi134 conmon[36363]: 2021-11-19T08:37 2021-11-19T08:38:01.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:00 smithi134 conmon[36363]: :59.628698+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:38:01.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:00 smithi134 conmon[36363]: .14164) 84 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-11-19T08:38:01.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:00 smithi134 conmon[36363]: 2021-11-19T08:38:02.838 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:02 smithi023 conmon[33490]: cluster 2021-11-19T08:38:00.800487+0000 mgr.smithi023.ehlpcl (mgr.14164) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:03.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:02 smithi134 conmon[36363]: cluster 2021-11-19T08:38:00.800487+0000 mgr.smithi023.ehlpcl (mgr.14164) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:03.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:02 smithi134 conmon[36363]: 2021-11-19T08:38:04.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:04 smithi023 conmon[33490]: cluster 2021-11-19T08:38:02.800803 2021-11-19T08:38:04.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:04 smithi023 conmon[33490]: +0000 mgr.smithi023.ehlpcl (mgr.14164) 86 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:04.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:04 smithi023 conmon[33490]: audit 2021-11-19T08:38:04.479983+0000 mon.smithi023 (mon.0) 250 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-11-19T08:38:04.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:04 smithi023 conmon[33490]: audit 2021-11-19T08:38:04.480597+0000 mon.smithi023 (mon.0) 251 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:05.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:04 smithi134 conmon[36363]: cluster 2021-11-19T08:38:02.800803+0000 mgr.smithi023.ehlpcl (mgr.14164) 86 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:05.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:04 smithi134 conmon[36363]: audit 2021-11-19T08:38:04 2021-11-19T08:38:05.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:04 smithi134 conmon[36363]: .479983+0000 mon.smithi023 (mon.0) 250 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-11-19T08:38:05.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:04 smithi134 conmon[36363]: audit 2021-11-19T08:38:04.480597+0000 mon.smithi023 (mon.0) 251 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:05.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:05 smithi023 conmon[33490]: cephadm 2021-11-19T08:38:04.481105+0000 mgr.smithi023.ehlpcl (mgr.14164) 87 : cephadm [INF] Deploying daemon osd.1 on smithi023 2021-11-19T08:38:06.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:05 smithi134 conmon[36363]: cephadm 2021-11-19T08:38:04.481105+0000 mgr.smithi023.ehlpcl (mgr.14164) 87 : cephadm [INF] Deploying daemon osd.1 on smithi023 2021-11-19T08:38:06.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:06 smithi023 conmon[33490]: cluster 2021-11-19T08:38:04.801037+0000 mgr.smithi023.ehlpcl (mgr.14164) 88 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:07.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:06 smithi134 conmon[36363]: cluster 2021-11-19T08:38:04.801037+0000 mgr.smithi023.ehlpcl (mgr.14164) 88 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:07.402 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 1 on host 'smithi023' 2021-11-19T08:38:07.405 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:07.414+0000 7fae25ffb700 1 -- 172.21.15.23:0/853764501 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fae200092e0 con 0x7fae1c05c2b0 2021-11-19T08:38:07.405 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:07.417+0000 7fae372f3700 1 -- 172.21.15.23:0/853764501 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fae1c05c2b0 msgr2=0x7fae1c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:38:07.405 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:07.417+0000 7fae372f3700 1 --2- 172.21.15.23:0/853764501 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fae1c05c2b0 0x7fae1c05e770 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fae20009d60 tx=0x7fae20009b30).stop 2021-11-19T08:38:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:07.418+0000 7fae372f3700 1 -- 172.21.15.23:0/853764501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae300f6860 msgr2=0x7fae3010e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:38:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:07.418+0000 7fae372f3700 1 --2- 172.21.15.23:0/853764501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae300f6860 0x7fae3010e5b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fae2c00c790 tx=0x7fae2c007d70).stop 2021-11-19T08:38:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:07.418+0000 7fae372f3700 1 -- 172.21.15.23:0/853764501 shutdown_connections 2021-11-19T08:38:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:07.418+0000 7fae372f3700 1 --2- 172.21.15.23:0/853764501 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae300f55c0 0x7fae3010c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:07.418+0000 7fae372f3700 1 --2- 172.21.15.23:0/853764501 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fae1c05c2b0 0x7fae1c05e770 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:07.418+0000 7fae372f3700 1 --2- 172.21.15.23:0/853764501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae300f6860 0x7fae3010e5b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:07.418+0000 7fae372f3700 1 -- 172.21.15.23:0/853764501 >> 172.21.15.23:0/853764501 conn(0x7fae300f0d40 msgr2=0x7fae300fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:38:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:07.418+0000 7fae372f3700 1 -- 172.21.15.23:0/853764501 shutdown_connections 2021-11-19T08:38:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:07.418+0000 7fae372f3700 1 -- 172.21.15.23:0/853764501 wait complete. 2021-11-19T08:38:07.745 DEBUG:teuthology.orchestra.run.smithi023:osd.1> sudo journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.1.service 2021-11-19T08:38:07.748 INFO:tasks.cephadm:Deploying osd.2 on smithi023 with /dev/vg_nvme/lv_2... 2021-11-19T08:38:07.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-11-19T08:38:07.779 INFO:journalctl@ceph.osd.1.smithi023.stdout:-- Logs begin at Fri 2021-11-19 08:24:48 UTC. -- 2021-11-19T08:38:08.142 INFO:journalctl@ceph.osd.1.smithi023.stdout:Nov 19 08:38:07 smithi023 conmon[55599]: debug 2021-11-19T08:38:07.866+0000 7f02e7044080 -1 Falling back to public interface 2021-11-19T08:38:08.484 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: cluster 2021-11-19T08:38 2021-11-19T08:38:08.484 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: :06.801277+0000 mgr.smithi023.ehlpcl (mgr.14164) 89 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:08.484 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.407321+0000 mon.smithi023 (mon.0) 252 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:08.485 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.415117+0000 mon.smithi023 (mon.0) 253 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:08.485 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.428934+0000 mon.smithi023 (mon.0) 254 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.23:9093"}]: dispatch 2021-11-19T08:38:08.485 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.429385+0000 mgr.smithi023.ehlpcl (mgr.14164) 90 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.23:9093"}]: dispatch 2021-11-19T08:38:08.485 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.434278+0000 mon.smithi023 (mon.0) 255 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:08.486 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.438092+0000 mon.smithi023 (mon.0) 256 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-19T08:38:08.486 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.438443+0000 mgr.smithi023.ehlpcl (mgr.14164) 91 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-19T08:38:08.486 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.445878+0000 mon.smithi023 (mon.0) 257 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.23:3000"}]: dispatch 2021-11-19T08:38:08.486 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.446336+0000 mgr.smithi023.ehlpcl (mgr.14164) 92 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.23:3000"}]: dispatch 2021-11-19T08:38:08.487 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.450417+0000 mon.smithi023 (mon.0) 258 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:08.487 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.454757+0000 mon.smithi023 (mon.0) 259 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-19T08:38:08.487 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.455191+0000 mgr.smithi023.ehlpcl (mgr.14164) 93 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-19T08:38:08.488 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.460326+0000 mon.smithi023 (mon.0) 260 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.23:9095"}]: dispatch 2021-11-19T08:38:08.488 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.460708+0000 mgr.smithi023.ehlpcl (mgr.14164) 94 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.23:9095"}]: dispatch 2021-11-19T08:38:08.488 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:07.464976+0000 mon.smithi023 (mon.0) 261 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:08.488 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: audit 2021-11-19T08:38:08.489 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:08 smithi023 conmon[33490]: 2021-11-19T08:38:07.468116+0000 mon.smithi023 (mon.0) 262 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:38:08.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: cluster 2021-11-19T08:38:06.801277+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:38:08.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: 14164) 89 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:08.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.407321+0000 mon.smithi023 (mon.0) 252 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:08.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.415117+0000 mon.smithi023 (mon.0) 253 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:08.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.428934+0000 mon.smithi023 (mon.0) 254 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.23:9093"}]: dispatch 2021-11-19T08:38:08.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.429385+0000 mgr.smithi023.ehlpcl (mgr.14164) 90 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.23:9093"}]: dispatch 2021-11-19T08:38:08.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.434278+0000 mon.smithi023 (mon.0) 255 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:08.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.438092+0000 mon.smithi023 (mon.0) 256 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-19T08:38:08.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.438443+0000 mgr.smithi023.ehlpcl (mgr.14164) 91 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-11-19T08:38:08.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.445878+0000 mon.smithi023 (mon.0) 257 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.23:3000"}]: dispatch 2021-11-19T08:38:08.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.446336+0000 mgr.smithi023.ehlpcl (mgr.14164) 92 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.23:3000"}]: dispatch 2021-11-19T08:38:08.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.450417+0000 mon.smithi023 (mon.0) 258 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:08.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.454757+0000 mon.smithi023 (mon.0) 259 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-19T08:38:08.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.455191+0000 mgr.smithi023.ehlpcl (mgr.14164) 93 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-11-19T08:38:08.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.460326+0000 mon.smithi023 (mon.0) 260 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.23:9095"}]: dispatch 2021-11-19T08:38:08.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.460708+0000 mgr.smithi023.ehlpcl (mgr.14164) 94 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.23:9095"}]: dispatch 2021-11-19T08:38:08.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.464976+0000 mon.smithi023 (mon.0) 261 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:08.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:08 smithi134 conmon[36363]: audit 2021-11-19T08:38:07.468116+0000 mon.smithi023 (mon.0) 262 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:38:09.921 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-11-19T08:38:09.921 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-11-19T08:38:09.922 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2021-11-19T08:38:09.922 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2021-11-19T08:38:09.922 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112027 s, 936 MB/s 2021-11-19T08:38:09.922 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2021-11-19T08:38:10.310 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi023:vg_nvme/lv_2 2021-11-19T08:38:10.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:10 smithi023 conmon[33490]: cluster 2021-11-19T08:38:08.801692+0000 mgr.smithi023.ehlpcl (mgr.14164) 95 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:10.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:10 smithi023 conmon[33490]: audit 2021-11-19T08:38:09.632134+0000 mon.smithi023 (mon.0) 263 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:11.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:10 smithi134 conmon[36363]: cluster 2021-11-19T08:38:08.801692+0000 mgr.smithi023.ehlpcl (mgr.14164) 95 : cluster 2021-11-19T08:38:11.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:10 smithi134 conmon[36363]: [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:11.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:10 smithi134 conmon[36363]: audit 2021-11-19T08:38:09.632134+0000 mon.smithi023 (mon.0) 263 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:11.617 INFO:journalctl@ceph.osd.1.smithi023.stdout:Nov 19 08:38:11 smithi023 conmon[55599]: debug 2021-11-19T08:38:11.375+0000 7f02e7044080 -1 osd.1 0 log_to_monitors {default=true} 2021-11-19T08:38:11.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:11 smithi023 conmon[33490]: audit 2021-11-19T08:38:11.379435+0000 mon.smithi023 (mon.0 2021-11-19T08:38:11.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:11 smithi023 conmon[33490]: ) 264 : audit [INF] from='osd.1 [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-11-19T08:38:12.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:11 smithi134 conmon[36363]: audit 2021-11-19T08:38:11.379435+0000 2021-11-19T08:38:12.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:11 smithi134 conmon[36363]: mon.smithi023 (mon.0) 264 : audit [INF] from='osd.1 [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-11-19T08:38:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.372+0000 7f177c88e700 1 -- 172.21.15.23:0/738638588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17780f6530 msgr2=0x7f17780f6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:38:12.360 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.372+0000 7f177c88e700 1 --2- 172.21.15.23:0/738638588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17780f6530 0x7f17780f6950 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f1768004660 tx=0x7f1768009b30).stop 2021-11-19T08:38:12.360 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.372+0000 7f177c88e700 1 -- 172.21.15.23:0/738638588 shutdown_connections 2021-11-19T08:38:12.360 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.372+0000 7f177c88e700 1 --2- 172.21.15.23:0/738638588 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17780f77d0 0x7f17780f7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-19T08:38:12.360 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.372+0000 7f177c88e700 1 --2- 172.21.15.23:0/738638588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17780f6530 0x7f17780f6950 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:12.361 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.372+0000 7f177c88e700 1 -- 172.21.15.23:0/738638588 >> 172.21.15.23:0/738638588 conn(0x7f17780f1c90 msgr2=0x7f17780f40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:38:12.361 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.372+0000 7f177c88e700 1 -- 172.21.15.23:0/738638588 shutdown_connections 2021-11-19T08:38:12.361 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.372+0000 7f177c88e700 1 -- 172.21.15.23:0/738638588 wait complete. 2021-11-19T08:38:12.361 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.372+0000 7f177c88e700 1 Processor -- start 2021-11-19T08:38:12.362 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.372+0000 7f177c88e700 1 -- start start 2021-11-19T08:38:12.362 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f177c88e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17780f6530 0x7f1778106e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:38:12.362 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f177c88e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17780f77d0 0x7f1778107330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:38:12.362 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f177c88e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17781079c0 con 0x7f17780f6530 2021-11-19T08:38:12.362 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f177c88e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1778108140 con 0x7f17780f77d0 2021-11-19T08:38:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f1776d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17780f6530 0x7f1778106e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:38:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f1776d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17780f6530 0x7f1778106e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46484/0 (socket says 172.21.15.23:46484) 2021-11-19T08:38:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f1776d9d700 1 -- 172.21.15.23:0/1059563696 learned_addr learned my addr 172.21.15.23:0/1059563696 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:38:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f177659c700 1 --2- 172.21.15.23:0/1059563696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17780f77d0 0x7f1778107330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:38:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f1776d9d700 1 -- 172.21.15.23:0/1059563696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17780f77d0 msgr2=0x7f1778107330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:38:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f1776d9d700 1 --2- 172.21.15.23:0/1059563696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17780f77d0 0x7f1778107330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f1776d9d700 1 -- 172.21.15.23:0/1059563696 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1768005040 con 0x7f17780f6530 2021-11-19T08:38:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f177659c700 1 --2- 172.21.15.23:0/1059563696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17780f77d0 0x7f1778107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:38:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f1776d9d700 1 --2- 172.21.15.23:0/1059563696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17780f6530 0x7f1778106e40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f1768009f50 tx=0x7f1768009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:38:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f176f7fe700 1 -- 172.21.15.23:0/1059563696 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1768018070 con 0x7f17780f6530 2021-11-19T08:38:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f176f7fe700 1 -- 172.21.15.23:0/1059563696 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1768013410 con 0x7f17780f6530 2021-11-19T08:38:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.373+0000 7f176f7fe700 1 -- 172.21.15.23:0/1059563696 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f176801c620 con 0x7f17780f6530 2021-11-19T08:38:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.374+0000 7f177c88e700 1 -- 172.21.15.23:0/1059563696 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1778108360 con 0x7f17780f6530 2021-11-19T08:38:12.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.374+0000 7f177c88e700 1 -- 172.21.15.23:0/1059563696 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17781088c0 con 0x7f17780f6530 2021-11-19T08:38:12.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.375+0000 7f177c88e700 1 -- 172.21.15.23:0/1059563696 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1778045ad0 con 0x7f17780f6530 2021-11-19T08:38:12.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.375+0000 7f176f7fe700 1 -- 172.21.15.23:0/1059563696 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f176801d420 con 0x7f17780f6530 2021-11-19T08:38:12.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.375+0000 7f176f7fe700 1 --2- 172.21.15.23:0/1059563696 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f176405c300 0x7f176405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:38:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.375+0000 7f176f7fe700 1 -- 172.21.15.23:0/1059563696 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2018+0+0 (secure 0 0 0) 0x7f1768077010 con 0x7f17780f6530 2021-11-19T08:38:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.375+0000 7f177659c700 1 --2- 172.21.15.23:0/1059563696 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f176405c300 0x7f176405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:38:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.376+0000 7f177659c700 1 --2- 172.21.15.23:0/1059563696 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f176405c300 0x7f176405e7c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f176000b830 tx=0x7f1760008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:38:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.379+0000 7f176f7fe700 1 -- 172.21.15.23:0/1059563696 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f176804a9f0 con 0x7f17780f6530 2021-11-19T08:38:12.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:12.575+0000 7f177c88e700 1 -- 172.21.15.23:0/1059563696 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f17780feef0 con 0x7f176405c300 2021-11-19T08:38:12.629 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:12 smithi023 conmon[33490]: cluster 2021-11-19T08:38:10.801995+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:38:12.892 INFO:journalctl@ceph.osd.1.smithi023.stdout:Nov 19 08:38:12 smithi023 conmon[55599]: debug 2021-11-19T08:38:12.645+0000 7f02dd5b5700 -1 osd.1 0 waiting for initial osdmap 2021-11-19T08:38:12.892 INFO:journalctl@ceph.osd.1.smithi023.stdout:Nov 19 08:38:12 smithi023 conmon[55599]: debug 2021-11-19T08:38:12.650+0000 7f02d852e700 -1 osd.1 13 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2021-11-19T08:38:12.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:12 smithi023 conmon[33490]: mgr.14164) 96 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:12.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:12 smithi023 conmon[33490]: audit 2021-11-19T08:38:11.640462+0000 mon.smithi023 (mon.0) 265 : audit [INF] from='osd.1 [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-11-19T08:38:12.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:12 smithi023 conmon[33490]: cluster 2021-11-19T08:38:11.640521+0000 mon.smithi023 (mon.0) 266 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-11-19T08:38:12.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:12 smithi023 conmon[33490]: audit 2021-11-19T08:38:11.640631+0000 mon.smithi023 (mon.0) 267 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-19T08:38:12.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:12 smithi023 conmon[33490]: audit 2021-11-19T08:38:11.640933+0000 mon.smithi023 (mon.0) 268 : audit [INF] from='osd.1 [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2021-11-19T08:38:12.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:12 smithi023 conmon[33490]: audit 2021-11-19T08:38:12.578276+0000 mon.smithi023 (mon.0) 269 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-19T08:38:12.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:12 smithi023 conmon[33490]: audit 2021-11-19T08:38:12.579739+0000 mon.smithi023 (mon.0) 270 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-19T08:38:12.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:12 smithi023 conmon[33490]: audit 2021-11-19T08:38:12.580341+0000 mon.smithi023 (mon.0) 271 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:13.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: cluster 2021-11-19T08:38:10.801995+0000 mgr.smithi023.ehlpcl (mgr.14164) 96 : cluster 2021-11-19T08:38:13.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:13.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: audit 2021-11-19T08:38:11.640462+0000 mon.smithi023 (mon.0) 265 : audit [INF] from='osd.1 [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-11-19T08:38:13.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: cluster 2021-11-19T08:38:13.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: 2021-11-19T08:38:11.640521+0000 mon.smithi023 (mon.0) 266 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-11-19T08:38:13.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: audit 2021-11- 2021-11-19T08:38:13.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: 19T08:38:11.640631+0000 mon.smithi023 (mon.0 2021-11-19T08:38:13.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: ) 267 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-19T08:38:13.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:38:13.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: 11-19T08:38:11.640933+0000 mon.smithi023 (mon 2021-11-19T08:38:13.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: .0) 268 : audit [INF] from='osd.1 [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2021-11-19T08:38:13.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:38:13.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: 11-19T08:38:12.578276+0000 mon.smithi023 (mon. 2021-11-19T08:38:13.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: 0) 269 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-19T08:38:13.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: audit 2021-11-19T08:38:13.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: 2021-11-19T08:38:12.579739+0000 2021-11-19T08:38:13.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: mon.smithi023 (mon.0) 270 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-19T08:38:13.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: 2021-11-19T08:38:13.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: audit 2021-11-19T08:38: 2021-11-19T08:38:13.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: 12.580341+0000 mon.smithi023 (mon.0) 271 2021-11-19T08:38:13.046 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:12 smithi134 conmon[36363]: : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:13.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:13 smithi023 conmon[33490]: audit 2021-11-19T08:38:12.577481+0000 mgr.smithi023.ehlpcl (mgr.14164) 97 : audit [DBG] from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:38:13.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:13 smithi023 conmon[33490]: cephadm 2021-11-19T08:38:12.579526+0000 mgr.smithi023.ehlpcl (mgr.14164) 98 : cephadm [INF] Applying service osd.None on host smithi023... 2021-11-19T08:38:13.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:13 smithi023 conmon[33490]: audit 2021-11-19T08:38:12.642036+0000 mon.smithi023 (mon.0) 272 : audit [INF] from='osd.1 [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2021-11-19T08:38:13.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:13 smithi023 conmon[33490]: cluster 2021-11-19T08:38:12.642093+0000 mon.smithi023 (mon.0) 273 : cluster [DBG] osdmap e13: 2 total, 1 up, 2 in 2021-11-19T08:38:13.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:13 smithi023 conmon[33490]: audit 2021-11-19T08:38:12.642836+0000 mon.smithi023 (mon.0) 274 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-19T08:38:13.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:13 smithi023 conmon[33490]: audit 2021-11-19T08:38:12.647177+0000 mon.smithi023 2021-11-19T08:38:13.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:13 smithi023 conmon[33490]: (mon.0) 275 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-19T08:38:13.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:13 smithi023 conmon[33490]: audit 2021-11-19T08:38:13.100365+0000 mon.smithi023 (mon.0) 276 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:14.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: audit 2021-11-19T08:38:12.577481+0000 mgr.smithi023.ehlpcl 2021-11-19T08:38:14.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: (mgr.14164) 97 : audit [DBG] from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:38:14.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: cephadm 2021-11-19T08:38:12.579526+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:38:14.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: ) 98 : cephadm [INF] Applying service osd.None on host smithi023... 2021-11-19T08:38:14.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: audit 2021-11-19T08:38:14.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: 2021-11-19T08:38:12.642036+0000 mon.smithi023 (mon.0) 272 : audit 2021-11-19T08:38:14.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: [INF] from='osd.1 [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2021-11-19T08:38:14.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: cluster 2021-11-19T08:38 2021-11-19T08:38:14.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: :12.642093+0000 mon.smithi023 (mon.0) 273 : cluster 2021-11-19T08:38:14.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: [DBG] osdmap e13: 2 total, 1 up, 2 in 2021-11-19T08:38:14.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: audit 2021-11-19T08 2021-11-19T08:38:14.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: :38:12.642836+0000 mon.smithi023 (mon.0) 274 2021-11-19T08:38:14.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-19T08:38:14.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: audit 2021-11-19T08 2021-11-19T08:38:14.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: :38:12.647177+0000 mon.smithi023 (mon.0) 2021-11-19T08:38:14.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: 275 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-19T08:38:14.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: audit 2021-11 2021-11-19T08:38:14.046 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: -19T08:38:13.100365+0000 mon.smithi023 ( 2021-11-19T08:38:14.046 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:13 smithi134 conmon[36363]: mon.0) 276 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:14.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:14 smithi023 conmon[33490]: cluster 2021-11-19T08:38:12.408126+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-19T08:38:14.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:14 smithi023 conmon[33490]: cluster 2021-11-19T08:38:12.408235+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-19T08:38:14.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:14 smithi023 conmon[33490]: cluster 2021-11-19T08:38:12.802232+0000 mgr.smithi023.ehlpcl (mgr.14164) 99 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:14.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:14 smithi023 conmon[33490]: audit 2021-11-19T08:38:13.645489+0000 mon.smithi023 (mon.0) 277 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-19T08:38:14.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:14 smithi023 conmon[33490]: cluster 2021-11-19T08:38:13.653800+0000 mon.smithi023 (mon.0) 278 : cluster [INF] osd.1 [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153] boot 2021-11-19T08:38:14.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:14 smithi023 conmon[33490]: cluster 2021-11-19T08:38:13.653829+0000 mon.smithi023 (mon.0) 279 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-11-19T08:38:14.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:14 smithi023 conmon[33490]: audit 2021-11-19T08:38:13.654300+0000 mon.smithi023 (mon.0) 280 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-19T08:38:15.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: cluster 2021-11-19T08:38:12.408126+0000 2021-11-19T08:38:15.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-19T08:38:15.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: cluster 2021-11-19T08:38:12.408235+0000 2021-11-19T08:38:15.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-19T08:38:15.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: cluster 2021-11-19T08: 2021-11-19T08:38:15.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: 38:12.802232+0000 mgr.smithi023.ehlpcl (mgr.14164) 99 : cluster 2021-11-19T08:38:15.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-11-19T08:38:15.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: audit 2021-11-19T 2021-11-19T08:38:15.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: 08:38:13.645489+0000 mon.smithi023 (mon.0 2021-11-19T08:38:15.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: ) 277 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-19T08:38:15.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: cluster 2021-11- 2021-11-19T08:38:15.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: 19T08:38:13.653800+0000 mon.smithi023 (mon.0 2021-11-19T08:38:15.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: ) 278 : cluster [INF] osd.1 [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153] boot 2021-11-19T08:38:15.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: cluster 2021 2021-11-19T08:38:15.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: -11-19T08:38:13.653829+0000 mon.smithi023 2021-11-19T08:38:15.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: (mon.0) 279 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-11-19T08:38:15.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:38:15.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: 11-19T08:38:13.654300+0000 mon.smithi023 (mon.0) 280 2021-11-19T08:38:15.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:14 smithi134 conmon[36363]: : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-11-19T08:38:16.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:16 smithi023 conmon[33490]: audit 2021-11-19T08:38:14.851550 2021-11-19T08:38:16.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:16 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 281 : audit [INF] from='client.? 172.21.15.23:0/2619365473' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e21cf6ec-4aa2-43cc-934a-bbd8d6e6694c"}]: dispatch 2021-11-19T08:38:16.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:16 smithi023 conmon[33490]: audit 2021-11-19T08:38:14.856398+0000 mon.smithi023 (mon.0) 282 : audit [INF] from='client.? 172.21.15.23:0/2619365473' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e21cf6ec-4aa2-43cc-934a-bbd8d6e6694c"}]': finished 2021-11-19T08:38:16.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:16 smithi023 conmon[33490]: cluster 2021-11-19T08:38:14.856443+0000 mon.smithi023 (mon.0) 283 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-11-19T08:38:16.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:16 smithi023 conmon[33490]: audit 2021-11-19T08:38:14.856534+0000 mon.smithi023 (mon.0) 284 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-19T08:38:16.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:16 smithi023 conmon[33490]: audit 2021-11-19T08:38:15.286656+0000 mon.smithi023 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.23:0/1666080330' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-19T08:38:16.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:16 smithi134 conmon[36363]: audit 2021-11-19T08:38:14.851550+0000 mon.smithi023 (mon.0) 281 : audit [INF] from='client.? 172.21.15.23:0/2619365473' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e21cf6ec-4aa2-43cc-934a-bbd8d6e6694c"}]: dispatch 2021-11-19T08:38:16.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:16 smithi134 conmon[36363]: audit 2021-11- 2021-11-19T08:38:16.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:16 smithi134 conmon[36363]: 19T08:38:14.856398+0000 mon.smithi023 (mon.0) 282 : audit [INF] from='client.? 172.21.15.23:0/2619365473' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e21cf6ec-4aa2-43cc-934a-bbd8d6e6694c"}]': finished 2021-11-19T08:38:16.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:16 smithi134 conmon[36363]: cluster 2021-11-19T08:38:14.856443+0000 mon.smithi023 (mon.0) 283 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-11-19T08:38:16.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:16 smithi134 conmon[36363]: audit 2021-11-19T08:38:14.856534+0000 mon.smithi023 2021-11-19T08:38:16.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:16 smithi134 conmon[36363]: (mon.0) 284 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-19T08:38:16.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:16 smithi134 conmon[36363]: audit 2021-11-19T 2021-11-19T08:38:16.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:16 smithi134 conmon[36363]: 08:38:15.286656+0000 mon.smithi023 (mon. 2021-11-19T08:38:16.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:16 smithi134 conmon[36363]: 0) 285 : audit [DBG] from='client.? 172.21.15.23:0/1666080330' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-19T08:38:17.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:17 smithi023 conmon[33490]: cluster 2021-11-19T08:38:14.802494+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:38:17.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:17 smithi023 conmon[33490]: ) 100 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:17.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:17 smithi134 conmon[36363]: cluster 2021-11-19T08:38:14.802494+0000 mgr.smithi023.ehlpcl (mgr.14164) 100 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:18.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:18 smithi023 conmon[33490]: cluster 2021-11-19T08:38:16.802901+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:38:18.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:18 smithi023 conmon[33490]: 101 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:18.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:18 smithi134 conmon[36363]: cluster 2021-11-19T08:38:16.802901+0000 mgr.smithi023.ehlpcl (mgr.14164) 101 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:20.205 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:20 smithi023 conmon[33490]: cluster 2021-11-19T08:38:18.803387+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:38:20.205 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:20 smithi023 conmon[33490]: ) 102 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:20.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:20 smithi134 conmon[36363]: cluster 2021-11-19T08:38:18.803387+0000 mgr.smithi023.ehlpcl (mgr.14164) 102 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:21.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:21 smithi023 conmon[33490]: audit 2021-11-19T08:38:20.911368+0000 mon.smithi023 (mon.0) 286 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:38:21.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:21 smithi023 conmon[33490]: audit 2021-11-19T08:38:20.911634+0000 mon.smithi023 (mon.0) 287 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:38:21.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:21 smithi134 conmon[36363]: audit 2021-11-19T08:38:20.911368+0000 mon.smithi023 (mon.0) 286 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:38:21.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:21 smithi134 conmon[36363]: audit 2021-11-19T08:38:21.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:21 smithi134 conmon[36363]: 2021-11-19T08:38:20.911634+0000 mon.smithi023 (mon.0) 287 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:38:22.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:22 smithi023 conmon[33490]: cluster 2021-11-19T08:38:20.803714+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:38:22.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:22 smithi023 conmon[33490]: 14164) 103 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:22.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:22 smithi134 conmon[36363]: cluster 2021-11-19T08:38:20.803714+0000 mgr.smithi023.ehlpcl (mgr.14164) 103 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:23.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: cephadm 2021-11-19T08:38:22.153353+0000 2021-11-19T08:38:23.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: mgr.smithi023.ehlpcl (mgr.14164) 104 : cephadm [INF] Detected new or changed devices on smithi023 2021-11-19T08:38:23.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: audit 2021-11-19T08:38:22.162123+0000 mon.smithi023 (mon. 2021-11-19T08:38:23.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: 0) 288 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:23.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: audit 2021-11-19 2021-11-19T08:38:23.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: T08:38:22.174067+0000 mon.smithi023 (mon.0) 289 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:23.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: audit 2021-11-19T08 2021-11-19T08:38:23.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: :38:22.177295+0000 mon.smithi023 (mon.0) 290 : 2021-11-19T08:38:23.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:38:23.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: audit 2021-11-19T08:38:22.864120+0000 2021-11-19T08:38:23.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: mon.smithi023 (mon.0) 291 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-11-19T08:38:23.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: audit 2021-11-19T08:38:23.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: 2021-11-19T08:38:22. 2021-11-19T08:38:23.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: 864804+0000 mon.smithi023 (mon.0) 292 : audit [DBG] 2021-11-19T08:38:23.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:23 smithi134 conmon[36363]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:23.582 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:23 smithi023 conmon[33490]: cephadm 2021-11-19T08:38:22.153353+0000 mgr.smithi023.ehlpcl (mgr.14164) 104 : cephadm [INF] Detected new or changed devices on smithi023 2021-11-19T08:38:23.582 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:23 smithi023 conmon[33490]: audit 2021-11-19T08:38:22.162123+0000 mon.smithi023 (mon.0) 288 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:23.582 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:23 smithi023 conmon[33490]: audit 2021-11-19T08:38:22.174067+0000 mon.smithi023 (mon.0) 289 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:23.583 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:23 smithi023 conmon[33490]: audit 2021-11-19T08:38:22.177295+0000 mon.smithi023 (mon.0) 290 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:38:23.583 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:23 smithi023 conmon[33490]: audit 2021-11-19T08:38:22.864120+0000 mon.smithi023 (mon.0) 291 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-11-19T08:38:23.583 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:23 smithi023 conmon[33490]: audit 2021-11-19T08:38:22.864804+0000 mon.smithi023 (mon.0) 292 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:24.508 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:24 smithi023 conmon[33490]: cluster 2021-11-19T08:38:22.803973+0000 2021-11-19T08:38:24.509 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:24 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 105 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:24.509 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:24 smithi023 conmon[33490]: cephadm 2021-11-19T08:38:22.865494+0000 mgr.smithi023.ehlpcl (mgr.14164) 106 : cephadm [INF] Deploying daemon osd.2 on smithi023 2021-11-19T08:38:24.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:24 smithi134 conmon[36363]: cluster 2021-11-19T08:38:22.803973+0000 mgr.smithi023.ehlpcl (mgr.14164) 105 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:24.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:24 smithi134 conmon[36363]: 2021-11-19T08:38:24.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:24 smithi134 conmon[36363]: cephadm 2021-11-19T08:38:22.865494+0000 mgr.smithi023.ehlpcl (mgr.14164) 106 : cephadm [INF] 2021-11-19T08:38:24.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:24 smithi134 conmon[36363]: Deploying daemon osd.2 on smithi023 2021-11-19T08:38:25.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:25 smithi134 conmon[36363]: audit 2021-11-19T08:38:24.227809+0000 mon.smithi023 (mon.0) 293 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:25.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:25 smithi023 conmon[33490]: audit 2021-11-19T08:38:24.227809+0000 mon.smithi023 (mon.0) 293 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:26.310 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:26 smithi023 conmon[33490]: cluster 2021-11-19T08:38:24.804329+0000 mgr.smithi023.ehlpcl (mgr.14164) 107 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:26.362 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 2 on host 'smithi023' 2021-11-19T08:38:26.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:26.375+0000 7f176f7fe700 1 -- 172.21.15.23:0/1059563696 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f176000b410 con 0x7f176405c300 2021-11-19T08:38:26.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:26.377+0000 7f177c88e700 1 -- 172.21.15.23:0/1059563696 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f176405c300 msgr2=0x7f176405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:38:26.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:26.377+0000 7f177c88e700 1 --2- 172.21.15.23:0/1059563696 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f176405c300 0x7f176405e7c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f176000b830 tx=0x7f1760008040).stop 2021-11-19T08:38:26.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:26.377+0000 7f177c88e700 1 -- 172.21.15.23:0/1059563696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17780f6530 msgr2=0x7f1778106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:38:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:26.377+0000 7f177c88e700 1 --2- 172.21.15.23:0/1059563696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17780f6530 0x7f1778106e40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f1768009f50 tx=0x7f1768009b30).stop 2021-11-19T08:38:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:26.377+0000 7f177c88e700 1 -- 172.21.15.23:0/1059563696 shutdown_connections 2021-11-19T08:38:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:26.377+0000 7f177c88e700 1 --2- 172.21.15.23:0/1059563696 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f176405c300 0x7f176405e7c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:26.377+0000 7f177c88e700 1 --2- 172.21.15.23:0/1059563696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17780f6530 0x7f1778106e40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:26.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:26.377+0000 7f177c88e700 1 --2- 172.21.15.23:0/1059563696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17780f77d0 0x7f1778107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:26.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:26.377+0000 7f177c88e700 1 -- 172.21.15.23:0/1059563696 >> 172.21.15.23:0/1059563696 conn(0x7f17780f1c90 msgr2=0x7f17780fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:38:26.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:26.377+0000 7f177c88e700 1 -- 172.21.15.23:0/1059563696 shutdown_connections 2021-11-19T08:38:26.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:38:26.378+0000 7f177c88e700 1 -- 172.21.15.23:0/1059563696 wait complete. 2021-11-19T08:38:26.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:26 smithi134 conmon[36363]: cluster 2021-11-19T08:38:24.804329+0000 mgr.smithi023.ehlpcl (mgr.14164) 107 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:26.870 DEBUG:teuthology.orchestra.run.smithi023:osd.2> sudo journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.2.service 2021-11-19T08:38:26.873 INFO:tasks.cephadm:Deploying osd.3 on smithi134 with /dev/vg_nvme/lv_4... 2021-11-19T08:38:26.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-11-19T08:38:26.902 INFO:journalctl@ceph.osd.2.smithi023.stdout:-- Logs begin at Fri 2021-11-19 08:24:48 UTC. -- 2021-11-19T08:38:27.572 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:27 smithi134 conmon[36363]: audit 2021-11-19T08:38:26.367630+0000 mon.smithi023 (mon.0) 294 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:27.572 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:27 smithi134 conmon[36363]: audit 2021-11-19T08:38:26.376266+0000 mon.smithi023 (mon.0) 295 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:27.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:27 smithi023 conmon[33490]: audit 2021-11-19T08:38:26.367630+0000 mon.smithi023 2021-11-19T08:38:27.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:27 smithi023 conmon[33490]: (mon.0) 294 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:27.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:27 smithi023 conmon[33490]: audit 2021-11-19T08:38:26.376266+0000 mon.smithi023 (mon.0) 295 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:28.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:28 smithi134 conmon[36363]: cluster 2021-11-19T08:38:26.804696+0000 mgr.smithi023.ehlpcl (mgr.14164) 108 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:28.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:28 smithi134 conmon[36363]: audit 2021-11-19T08:38:28.104296+0000 mon.smithi023 (mon.0) 296 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:28.636 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:28 smithi023 conmon[33490]: cluster 2021-11-19T08:38:26.804696+0000 mgr.smithi023.ehlpcl (mgr.14164) 108 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:28.636 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:28 smithi023 conmon[33490]: audit 2021-11-19T08:38:28.104296+0000 mon.smithi023 (mon.0) 296 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:28.705 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-11-19T08:38:28.705 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-11-19T08:38:28.706 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2021-11-19T08:38:28.706 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2021-11-19T08:38:28.706 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109911 s, 954 MB/s 2021-11-19T08:38:28.706 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2021-11-19T08:38:29.087 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi134:vg_nvme/lv_4 2021-11-19T08:38:30.475 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:30 smithi134 conmon[36363]: cluster 2021-11-19T08:38:28.805046+0000 mgr.smithi023.ehlpcl (mgr.14164) 109 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:30.475 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:30 smithi134 conmon[36363]: audit 2021-11-19T08:38:30.271043+0000 mon.smithi023 (mon.0) 297 : audit [INF] from='osd.2 [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-11-19T08:38:30.605 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:30 smithi023 conmon[33490]: cluster 2021-11-19T08:38:28.805046+0000 mgr.smithi023.ehlpcl (mgr.14164) 109 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:30.605 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:30 smithi023 conmon[33490]: audit 2021-11-19T08:38:30.271043+0000 mon.smithi023 (mon.0) 297 : audit [INF] from='osd.2 [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-11-19T08:38:30.605 INFO:journalctl@ceph.osd.2.smithi023.stdout:Nov 19 08:38:30 smithi023 conmon[61849]: debug 2021-11-19T08:38:30.266+0000 7f98f7411080 -1 osd.2 0 log_to_monitors {default=true} 2021-11-19T08:38:30.643 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.656+0000 7ff78b59e700 1 -- 172.21.15.134:0/1562342842 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f4800 msgr2=0x7ff78c0f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:38:30.644 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.656+0000 7ff78b59e700 1 --2- 172.21.15.134:0/1562342842 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f4800 0x7ff78c0f4c20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7ff77c0099e0 tx=0x7ff77c004d10).stop 2021-11-19T08:38:30.644 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.656+0000 7ff78b59e700 1 -- 172.21.15.134:0/1562342842 shutdown_connections 2021-11-19T08:38:30.644 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.656+0000 7ff78b59e700 1 --2- 172.21.15.134:0/1562342842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f51a0 0x7ff78c0f7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:30.645 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.656+0000 7ff78b59e700 1 --2- 172.21.15.134:0/1562342842 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f4800 0x7ff78c0f4c20 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:30.645 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.656+0000 7ff78b59e700 1 -- 172.21.15.134:0/1562342842 >> 172.21.15.134:0/1562342842 conn(0x7ff78c0f0320 msgr2=0x7ff78c0f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:38:30.645 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.656+0000 7ff78b59e700 1 -- 172.21.15.134:0/1562342842 shutdown_connections 2021-11-19T08:38:30.645 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.656+0000 7ff78b59e700 1 -- 172.21.15.134:0/1562342842 wait complete. 2021-11-19T08:38:30.645 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.657+0000 7ff78b59e700 1 Processor -- start 2021-11-19T08:38:30.646 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.657+0000 7ff78b59e700 1 -- start start 2021-11-19T08:38:30.646 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.657+0000 7ff78b59e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f4800 0x7ff78c106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:38:30.646 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.657+0000 7ff78b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f51a0 0x7ff78c1073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:38:30.646 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.657+0000 7ff78b59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c107980 con 0x7ff78c0f51a0 2021-11-19T08:38:30.647 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.657+0000 7ff78b59e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c107ac0 con 0x7ff78c0f4800 2021-11-19T08:38:30.647 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.657+0000 7ff78a59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f4800 0x7ff78c106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:38:30.647 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.657+0000 7ff78a59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f4800 0x7ff78c106ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47772/0 (socket says 172.21.15.134:47772) 2021-11-19T08:38:30.647 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.657+0000 7ff78a59c700 1 -- 172.21.15.134:0/1520560142 learned_addr learned my addr 172.21.15.134:0/1520560142 (peer_addr_for_me v2:172.21.15.134:0/0) 2021-11-19T08:38:30.648 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.657+0000 7ff789d9b700 1 --2- 172.21.15.134:0/1520560142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f51a0 0x7ff78c1073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:38:30.648 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.657+0000 7ff78a59c700 1 -- 172.21.15.134:0/1520560142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f51a0 msgr2=0x7ff78c1073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:38:30.648 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.657+0000 7ff78a59c700 1 --2- 172.21.15.134:0/1520560142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f51a0 0x7ff78c1073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:30.648 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.657+0000 7ff78a59c700 1 -- 172.21.15.134:0/1520560142 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c009640 con 0x7ff78c0f4800 2021-11-19T08:38:30.648 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.658+0000 7ff78a59c700 1 --2- 172.21.15.134:0/1520560142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f4800 0x7ff78c106ee0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7ff77c0078b0 tx=0x7ff77c004d10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:38:30.649 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.658+0000 7ff77affd700 1 -- 172.21.15.134:0/1520560142 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff77c00e030 con 0x7ff78c0f4800 2021-11-19T08:38:30.649 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.658+0000 7ff77affd700 1 -- 172.21.15.134:0/1520560142 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff77c007d50 con 0x7ff78c0f4800 2021-11-19T08:38:30.649 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.658+0000 7ff77affd700 1 -- 172.21.15.134:0/1520560142 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff77c018070 con 0x7ff78c0f4800 2021-11-19T08:38:30.649 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.658+0000 7ff78b59e700 1 -- 172.21.15.134:0/1520560142 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78c108400 con 0x7ff78c0f4800 2021-11-19T08:38:30.649 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.658+0000 7ff78b59e700 1 -- 172.21.15.134:0/1520560142 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c1088e0 con 0x7ff78c0f4800 2021-11-19T08:38:30.650 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.659+0000 7ff78b59e700 1 -- 172.21.15.134:0/1520560142 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff78c045ad0 con 0x7ff78c0f4800 2021-11-19T08:38:30.651 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.659+0000 7ff77affd700 1 -- 172.21.15.134:0/1520560142 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff77c01d420 con 0x7ff78c0f4800 2021-11-19T08:38:30.651 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.660+0000 7ff77affd700 1 --2- 172.21.15.134:0/1520560142 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff77405c300 0x7ff77405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:38:30.652 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.660+0000 7ff77affd700 1 -- 172.21.15.134:0/1520560142 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2450+0+0 (secure 0 0 0) 0x7ff77c077050 con 0x7ff78c0f4800 2021-11-19T08:38:30.652 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.660+0000 7ff789d9b700 1 --2- 172.21.15.134:0/1520560142 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff77405c300 0x7ff77405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:38:30.652 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.660+0000 7ff789d9b700 1 --2- 172.21.15.134:0/1520560142 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff77405c300 0x7ff77405e7c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7ff78c0f55e0 tx=0x7ff78000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:38:30.652 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.664+0000 7ff77affd700 1 -- 172.21.15.134:0/1520560142 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff77c04a900 con 0x7ff78c0f4800 2021-11-19T08:38:30.825 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:30.840+0000 7ff78b59e700 1 -- 172.21.15.134:0/1520560142 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff78c0fedf0 con 0x7ff77405c300 2021-11-19T08:38:31.528 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: audit 2021-11-19T08:38:30.392302+0000 mon.smithi023 (mon 2021-11-19T08:38:31.528 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: .0) 298 : audit [INF] from='osd.2 [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-11-19T08:38:31.528 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: cluster 2021-11-19T08:38:30.392348+0000 mon.smithi023 (mon.0) 299 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-11-19T08:38:31.529 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: audit 2021-11-19T08:38:30.392786+0000 mon.smithi023 (mon.0) 300 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-19T08:38:31.529 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: audit 2021-11-19T08:38:30.393039+0000 mon.smithi023 (mon.0) 301 : audit [INF] from='osd.2 [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2021-11-19T08:38:31.529 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: audit 2021-11-19T08:38:30.842858+0000 mon.smithi023 (mon.0) 302 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-19T08:38:31.529 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: audit 2021-11-19T08:38:30.844376+0000 mon.smithi023 (mon.0) 303 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-19T08:38:31.529 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: audit 2021-11-19T08:38:30.844977+0000 mon.smithi023 (mon.0) 304 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:31.530 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: audit 2021-11-19T08:38:31.130903+0000 mon.smithi023 (mon.0) 305 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:31.530 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: audit 2021-11-19T08:38:31.136222+0000 mon.smithi023 (mon.0) 306 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:31.530 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: audit 2021-11-19T08:38:31.138925+0000 mon.smithi023 (mon.0) 307 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:38:31.530 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: audit 2021-11- 2021-11-19T08:38:31.531 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: 19T08:38:31.393486+0000 mon.smithi023 (mon.0) 308 : audit [INF] from='osd.2 [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2021-11-19T08:38:31.531 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: cluster 2021-11-19T08:38:31.393543+0000 mon.smithi023 (mon.0) 309 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-11-19T08:38:31.531 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: audit 2021-11-19T08:38 2021-11-19T08:38:31.531 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:31 smithi134 conmon[36363]: :31.393642+0000 mon.smithi023 (mon.0) 310 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-19T08:38:31.642 INFO:journalctl@ceph.osd.2.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[61849]: debug 2021-11-19T08:38:31.394+0000 7f98ed982700 -1 osd.2 0 waiting for initial osdmap 2021-11-19T08:38:31.642 INFO:journalctl@ceph.osd.2.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[61849]: debug 2021-11-19T08:38:31.401+0000 7f98e78f9700 -1 osd.2 17 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2021-11-19T08:38:31.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: audit 2021-11-19T08:38 2021-11-19T08:38:31.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: :30.392302+0000 mon.smithi023 (mon.0) 298 : audit [INF] from='osd.2 [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-11-19T08:38:31.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:38:31.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: 08:38:30.392348+0000 2021-11-19T08:38:31.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: mon.smithi023 (mon.0) 299 2021-11-19T08:38:31.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-11-19T08:38:31.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:38:31.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: -19T08:38:30. 2021-11-19T08:38:31.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: 392786+0000 mon.smithi023 (mon.0 2021-11-19T08:38:31.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: ) 300 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-19T08:38:31.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: audit 2021 2021-11-19T08:38:31.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: -11-19T08:38: 2021-11-19T08:38:31.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: 30.393039+0000 mon.smithi023 (mon 2021-11-19T08:38:31.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: .0) 301 : audit [INF] 2021-11-19T08:38:31.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: from='osd.2 [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2021-11-19T08:38:31.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: audit 2021-11-19T08:38:31.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: 2021-11-19T08: 2021-11-19T08:38:31.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: 38:30.842858+0000 2021-11-19T08:38:31.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: mon.smithi023 (mon.0) 302 : 2021-11-19T08:38:31.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-19T08:38:31.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: audit 2021-11-19T08:38:31.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: 2021-11-19T08 2021-11-19T08:38:31.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: :38:30.844376+0000 2021-11-19T08:38:31.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: mon.smithi023 (mon.0) 303 2021-11-19T08:38:31.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-19T08:38:31.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: audit 2021 2021-11-19T08:38:31.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: -11-19T08:38: 2021-11-19T08:38:31.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: 30.844977+0000 mon.smithi023 ( 2021-11-19T08:38:31.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: mon.0) 304 : audit 2021-11-19T08:38:31.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:31.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: audit 2021 2021-11-19T08:38:31.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: -11-19T08: 2021-11-19T08:38:31.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: 38:31.130903+0000 2021-11-19T08:38:31.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: mon.smithi023 (mon.0) 305 : 2021-11-19T08:38:31.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:31.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: audit 2021 2021-11-19T08:38:31.652 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: -11-19T08:38: 2021-11-19T08:38:31.652 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: 31.136222+0000 mon.smithi023 (mon 2021-11-19T08:38:31.652 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: .0) 306 : audit [INF] 2021-11-19T08:38:31.652 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:31.652 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: audit 2021-11-19T08:38:31.653 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: 2021-11-19T08 2021-11-19T08:38:31.653 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: :38:31.138925+0000 2021-11-19T08:38:31.653 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: mon.smithi023 (mon.0) 307 2021-11-19T08:38:31.653 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:38:31.654 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: audit 2021-11-19T08:38:31.654 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: 2021-11-19T08:38 2021-11-19T08:38:31.654 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: :31.393486+0000 mon.smithi023 (mon 2021-11-19T08:38:31.654 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: .0) 308 : audit [INF] 2021-11-19T08:38:31.654 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: from='osd.2 [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2021-11-19T08:38:31.655 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: cluster 2021 2021-11-19T08:38:31.656 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: -11-19T08:38:31.393543+0000 2021-11-19T08:38:31.656 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: mon.smithi023 (mon.0) 309 : 2021-11-19T08:38:31.656 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-11-19T08:38:31.656 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: audit 2021-11-19T08:38:31.657 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: 2021-11-19T08:38:31.393642 2021-11-19T08:38:31.657 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 310 : 2021-11-19T08:38:31.657 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:31 smithi023 conmon[33490]: audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-19T08:38:32.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:32 smithi134 conmon[36363]: cluster 2021-11-19T08:38:30.805337+0000 mgr.smithi023.ehlpcl (mgr.14164) 110 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:32.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:32 smithi134 conmon[36363]: audit 2021-11-19T08:38:30.841860+0000 mgr.smithi023.ehlpcl (mgr.14164) 111 : audit [DBG] from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:38:32.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:32 smithi134 conmon[36363]: cephadm 2021-11-19T08:38:30.844180+0000 mgr.smithi023.ehlpcl (mgr.14164) 112 : cephadm [INF] Applying service osd.None on host smithi134... 2021-11-19T08:38:32.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:32 smithi134 conmon[36363]: audit 2021-11-19T08:38:31.396615+0000 mon.smithi023 (mon.0) 311 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-19T08:38:32.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:32 smithi134 conmon[36363]: audit 2021-11-19T08:38:32.396505+0000 mon.smithi023 (mon.0) 312 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-19T08:38:32.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:32 smithi023 conmon[33490]: cluster 2021-11-19T08:38 2021-11-19T08:38:32.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:32 smithi023 conmon[33490]: :30.805337+0000 mgr.smithi023.ehlpcl (mgr.14164) 110 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:32.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:32 smithi023 conmon[33490]: audit 2021-11-19T08:38:30.841860+0000 mgr.smithi023.ehlpcl (mgr.14164) 111 : audit [DBG] from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:38:32.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:32 smithi023 conmon[33490]: cephadm 2021-11-19T08:38:30.844180+0000 mgr.smithi023.ehlpcl (mgr.14164) 112 : cephadm [INF] Applying service osd.None on host smithi134... 2021-11-19T08:38:32.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:32 smithi023 conmon[33490]: audit 2021-11-19T08:38:31.396615+0000 mon.smithi023 (mon.0) 311 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-19T08:38:32.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:32 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:38:32.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:32 smithi023 conmon[33490]: 11-19T08:38:32.396505+0000 mon.smithi023 (mon.0) 312 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-19T08:38:33.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:33 smithi134 conmon[36363]: cluster 2021-11-19T08:38:31.279858+0000 osd.2 (osd.2) 1 : cluster [DBG] 2021-11-19T08:38:33.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:33 smithi134 conmon[36363]: purged_snaps scrub starts 2021-11-19T08:38:33.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:33 smithi134 conmon[36363]: cluster 2021-11-19T08:38:31.280013+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-19T08:38:33.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:33 smithi134 conmon[36363]: cluster 2021-11-19T08:38:32.405822+0000 mon.smithi023 (mon.0) 313 : cluster [INF] osd.2 [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054] boot 2021-11-19T08:38:33.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:33 smithi134 conmon[36363]: cluster 2021-11-19T08:38:32.405849+0000 mon.smithi023 (mon.0) 314 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-11-19T08:38:33.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:33 smithi134 conmon[36363]: audit 2021-11-19T08:38:32.406319+0000 mon.smithi023 (mon.0) 315 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-19T08:38:33.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:33 smithi134 conmon[36363]: audit 2021-11-19T08:38:32.956494+0000 mon.smithi134 (mon.1) 2 : audit [INF] from='client.? 172.21.15.134:0/698929772' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "276cb410-5d23-4cdf-a090-e777ebe4c82f"}]: dispatch 2021-11-19T08:38:33.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:33 smithi134 conmon[36363]: audit 2021-11-19T08:38:32.956503+0000 mon.smithi023 (mon.0) 316 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "276cb410-5d23-4cdf-a090-e777ebe4c82f"}]: dispatch 2021-11-19T08:38:33.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:33 smithi134 conmon[36363]: audit 2021-11-19T08:38:32.964203+0000 mon.smithi023 (mon.0) 317 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "276cb410-5d23-4cdf-a090-e777ebe4c82f"}]': finished 2021-11-19T08:38:33.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:33 smithi134 conmon[36363]: cluster 2021-11-19T08:38:32.964263+0000 mon.smithi023 (mon.0) 318 : cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2021-11-19T08:38:33.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:33 smithi134 conmon[36363]: audit 2021-11-19T08:38:32.964358+0000 mon.smithi023 (mon.0) 319 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:33.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:33 smithi134 conmon[36363]: audit 2021-11-19T08:38:33.385844+0000 mon.smithi134 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.134:0/2751505543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-19T08:38:33.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: cluster 2021-11-19T08:38:31.279858+0000 2021-11-19T08:38:33.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-19T08:38:33.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: cluster 2021-11-19T08:38:31.280013+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-19T08:38:33.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: cluster 2021-11-19T08:38:32.405822+0000 mon.smithi023 (mon.0) 313 : cluster [INF] osd.2 [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054] boot 2021-11-19T08:38:33.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: cluster 2021-11-19T08:38:33.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: 2021-11-19T08:38:32.405849+0000 mon.smithi023 (mon.0) 314 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-11-19T08:38:33.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:38:33.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: 11-19T08:38:32.406319+0000 2021-11-19T08:38:33.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: mon.smithi023 (mon.0) 315 : audit 2021-11-19T08:38:33.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-11-19T08:38:33.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: audit 2021-11-19T08:38: 2021-11-19T08:38:33.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: 32.956494+0000 mon.smithi134 (mon.1) 2021-11-19T08:38:33.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: 2 : audit [INF] from='client.? 172.21.15.134:0/698929772' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "276cb410-5d23-4cdf-a090-e777ebe4c82f"}]: dispatch 2021-11-19T08:38:33.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: audit 2021-11-19T 2021-11-19T08:38:33.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: 08:38 2021-11-19T08:38:33.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: :32.956503+0000 mon.smithi023 ( 2021-11-19T08:38:33.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: mon.0) 316 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "276cb410-5d23-4cdf-a090-e777ebe4c82f"}]: dispatch 2021-11-19T08:38:33.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: 2021-11-19T08:38:33.898 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: audit 2021-11-19T08:38:33.898 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: 2021-11-19T08:38:32.964203+0000 2021-11-19T08:38:33.898 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: mon.smithi023 (mon.0) 317 : audit [INF] 2021-11-19T08:38:33.898 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "276cb410-5d23-4cdf-a090-e777ebe4c82f"}]': finished 2021-11-19T08:38:33.898 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: cluster 2021- 2021-11-19T08:38:33.899 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: 11-19T08:38:32.964263+0000 mon.smithi023 (mon 2021-11-19T08:38:33.899 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: .0) 318 : cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2021-11-19T08:38:33.899 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: audit 2021-11-19T08:38:32.964358+0000 mon.smithi023 2021-11-19T08:38:33.899 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: (mon.0) 319 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:33.900 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: audit 2021-11-19T08:38:33.385844 2021-11-19T08:38:33.900 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:33 smithi023 conmon[33490]: +0000 mon.smithi134 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.134:0/2751505543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-19T08:38:34.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:34 smithi134 conmon[36363]: cluster 2021-11-19T08:38:32.805619+0000 mgr.smithi023.ehlpcl (mgr.14164) 113 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:34.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:34 smithi134 conmon[36363]: audit 2021-11-19T08:38:33.829061+0000 mon.smithi023 (mon.0) 320 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:34.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:34 smithi134 conmon[36363]: cluster 2021-11-19T08:38:33.964841+0000 mon.smithi023 (mon.0) 321 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2021-11-19T08:38:34.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:34 smithi134 conmon[36363]: audit 2021-11-19T08:38:33.964976+0000 mon.smithi023 (mon.0) 322 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:34.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:34 smithi023 conmon[33490]: cluster 2021-11-19T08:38:32.805619+0000 mgr.smithi023.ehlpcl (mgr.14164) 113 : 2021-11-19T08:38:34.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:34 smithi023 conmon[33490]: cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-11-19T08:38:34.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:34 smithi023 conmon[33490]: audit 2021-11-19T08:38:33.829061+0000 mon.smithi023 (mon.0) 320 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:34.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:34 smithi023 conmon[33490]: cluster 2021-11-19T08:38:33.964841+0000 mon.smithi023 (mon 2021-11-19T08:38:34.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:34 smithi023 conmon[33490]: .0) 321 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2021-11-19T08:38:34.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:34 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:38:34.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:34 smithi023 conmon[33490]: T08:38:33.964976+0000 mon.smithi023 (mon.0) 322 : 2021-11-19T08:38:34.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:34 smithi023 conmon[33490]: audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:36.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:36 smithi134 conmon[36363]: cluster 2021-11-19T08:38:34.805878+0000 mgr.smithi023.ehlpcl (mgr.14164) 114 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:36.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:36 smithi134 conmon[36363]: audit 2021-11-19T08:38:35 2021-11-19T08:38:36.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:36 smithi134 conmon[36363]: .968798+0000 mon.smithi023 (mon.0) 323 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:36.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:36 smithi023 conmon[33490]: cluster 2021-11-19T08:38:34.805878+0000 mgr.smithi023.ehlpcl (mgr.14164) 114 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:36.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:36 smithi023 conmon[33490]: 2021-11-19T08:38:36.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:36 smithi023 conmon[33490]: audit 2021-11-19T08:38:35.968798+0000 mon.smithi023 (mon.0) 323 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:38.668 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:38 smithi134 conmon[36363]: cluster 2021-11-19T08:38:36.806133+0000 mgr.smithi023.ehlpcl (mgr.14164) 115 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:38.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:38 smithi023 conmon[33490]: cluster 2021-11-19T08:38:36.806133+0000 mgr.smithi023.ehlpcl (mgr.14164) 115 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:39.837 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:39 smithi134 conmon[36363]: audit 2021-11-19T08:38 2021-11-19T08:38:39.838 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:39 smithi134 conmon[36363]: :38.792319+0000 mon.smithi023 (mon.0) 324 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:39.838 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:39 smithi134 conmon[36363]: audit 2021-11-19T08:38:39.030378+0000 mon.smithi023 (mon.0) 325 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-11-19T08:38:39.839 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:39 smithi134 conmon[36363]: audit 2021-11-19T08:38:39.031007+0000 mon.smithi023 (mon.0) 326 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:39.839 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:39 smithi134 conmon[36363]: audit 2021-11-19T08:38:39.162687+0000 mon.smithi023 (mon.0) 327 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:40.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:39 smithi023 conmon[33490]: audit 2021-11-19T08:38:38.792319+0000 mon.smithi023 (mon.0) 2021-11-19T08:38:40.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:39 smithi023 conmon[33490]: 324 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:40.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:39 smithi023 conmon[33490]: audit 2021-11-19T08:38:39.030378+0000 mon.smithi023 (mon.0) 325 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-11-19T08:38:40.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:39 smithi023 conmon[33490]: audit 2021-11-19T08:38:39.031007+0000 mon.smithi023 (mon.0) 326 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:40.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:39 smithi023 conmon[33490]: audit 2021-11-19T08:38:39.162687+0000 mon.smithi023 (mon.0) 327 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:41.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:40 smithi134 conmon[36363]: cluster 2021-11-19T08:38:38.806374+0000 2021-11-19T08:38:41.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:40 smithi134 conmon[36363]: mgr.smithi023.ehlpcl (mgr.14164) 116 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:41.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:40 smithi134 conmon[36363]: cephadm 2021-11-19 2021-11-19T08:38:41.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:40 smithi134 conmon[36363]: T08:38:39.031533+0000 mgr.smithi023.ehlpcl (mgr.14164) 117 2021-11-19T08:38:41.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:40 smithi134 conmon[36363]: : cephadm [INF] Deploying daemon osd.3 on smithi134 2021-11-19T08:38:41.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:40 smithi023 conmon[33490]: cluster 2021-11-19T08:38:38.806374+0000 mgr.smithi023.ehlpcl (mgr.14164) 116 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:41.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:40 smithi023 conmon[33490]: 2021-11-19T08:38:41.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:40 smithi023 conmon[33490]: cephadm 2021-11-19T08:38:39.031533+0000 mgr.smithi023.ehlpcl (mgr.14164) 117 : cephadm [INF] Deploying daemon osd.3 on smithi134 2021-11-19T08:38:42.067 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:41 smithi134 conmon[36363]: cluster 2021-11-19T08:38:40.806826+0000 mgr.smithi023.ehlpcl (mgr.14164) 118 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:42.068 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:41 smithi134 conmon[36363]: 2021-11-19T08:38:42.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:41 smithi023 conmon[33490]: cluster 2021-11-19T08:38:40.806826+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:38:42.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:41 smithi023 conmon[33490]: mgr.14164) 118 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:42.210 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 3 on host 'smithi134' 2021-11-19T08:38:42.211 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:42.223+0000 7ff77affd700 1 -- 172.21.15.134:0/1520560142 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff78c0fedf0 con 0x7ff77405c300 2021-11-19T08:38:42.211 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:42.226+0000 7ff78b59e700 1 -- 172.21.15.134:0/1520560142 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff77405c300 msgr2=0x7ff77405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:38:42.212 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:42.226+0000 7ff78b59e700 1 --2- 172.21.15.134:0/1520560142 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff77405c300 0x7ff77405e7c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7ff78c0f55e0 tx=0x7ff78000b040).stop 2021-11-19T08:38:42.212 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:42.226+0000 7ff78b59e700 1 -- 172.21.15.134:0/1520560142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f4800 msgr2=0x7ff78c106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:38:42.212 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:42.226+0000 7ff78b59e700 1 --2- 172.21.15.134:0/1520560142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f4800 0x7ff78c106ee0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7ff77c0078b0 tx=0x7ff77c004d10).stop 2021-11-19T08:38:42.212 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:42.226+0000 7ff78b59e700 1 -- 172.21.15.134:0/1520560142 shutdown_connections 2021-11-19T08:38:42.213 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:42.226+0000 7ff78b59e700 1 --2- 172.21.15.134:0/1520560142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f4800 0x7ff78c106ee0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:42.213 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:42.226+0000 7ff78b59e700 1 --2- 172.21.15.134:0/1520560142 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff77405c300 0x7ff77405e7c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:42.213 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:42.226+0000 7ff78b59e700 1 --2- 172.21.15.134:0/1520560142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f51a0 0x7ff78c1073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:42.213 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:42.226+0000 7ff78b59e700 1 -- 172.21.15.134:0/1520560142 >> 172.21.15.134:0/1520560142 conn(0x7ff78c0f0320 msgr2=0x7ff78c0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:38:42.213 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:42.226+0000 7ff78b59e700 1 -- 172.21.15.134:0/1520560142 shutdown_connections 2021-11-19T08:38:42.214 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:42.226+0000 7ff78b59e700 1 -- 172.21.15.134:0/1520560142 wait complete. 2021-11-19T08:38:42.518 DEBUG:teuthology.orchestra.run.smithi134:osd.3> sudo journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.3.service 2021-11-19T08:38:42.521 INFO:tasks.cephadm:Deploying osd.4 on smithi134 with /dev/vg_nvme/lv_3... 2021-11-19T08:38:42.521 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-11-19T08:38:42.557 INFO:journalctl@ceph.osd.3.smithi134.stdout:-- Logs begin at Fri 2021-11-19 08:24:18 UTC. -- 2021-11-19T08:38:42.790 INFO:journalctl@ceph.osd.3.smithi134.stdout:Nov 19 08:38:42 smithi134 conmon[42726]: debug 2021-11-19T08:38:42.601+0000 7eff60732080 -1 Falling back to public interface 2021-11-19T08:38:43.101 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:42 smithi134 conmon[36363]: audit 2021-11-19T08:38:42.210324+0000 mon.smithi023 (mon.0) 328 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:38:43.101 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:42 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:38:43.102 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:42 smithi134 conmon[36363]: 11-19T08:38:42.216571+0000 mon.smithi023 (mon.0) 329 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:43.102 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:42 smithi134 conmon[36363]: audit 2021-11-19T08:38:42.223574+0000 mon.smithi023 (mon.0) 330 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:43.102 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:42 smithi134 conmon[36363]: audit 2021-11-19T08:38:42.227084+0000 mon.smithi023 (mon.0) 331 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-11-19T08:38:43.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:42 smithi023 conmon[33490]: audit 2021-11-19T08:38:42.210324+0000 mon.smithi023 (mon 2021-11-19T08:38:43.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:42 smithi023 conmon[33490]: .0) 328 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:38:43.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:42 smithi023 conmon[33490]: audit 2021-11-19T08:38:42.216571+0000 mon.smithi023 (mon.0) 329 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:43.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:42 smithi023 conmon[33490]: audit 2021-11-19T08:38:42.223574+0000 mon.smithi023 (mon.0) 330 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:43.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:42 smithi023 conmon[33490]: audit 2021-11-19T08:38:42.227084+0000 mon.smithi023 (mon.0) 331 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-11-19T08:38:44.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:44 smithi134 conmon[36363]: cluster 2021-11-19T08:38:42.807183+0000 mgr.smithi023.ehlpcl (mgr.14164) 119 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:44.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:44 smithi134 conmon[36363]: audit 2021-11-19T08:38:43.229599+0000 mon.smithi023 (mon.0) 332 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-11-19T08:38:44.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:44 smithi134 conmon[36363]: cluster 2021-11-19T08:38:43.229646+0000 mon.smithi023 (mon.0) 333 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2021-11-19T08:38:44.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:44 smithi134 conmon[36363]: audit 2021-11-19T08:38:43.229738+0000 mon.smithi023 (mon.0) 334 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:44.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:44 smithi134 conmon[36363]: audit 2021-11-19T08:38:43.230497+0000 mon.smithi023 (mon.0) 335 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-11-19T08:38:44.628 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-11-19T08:38:44.629 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-11-19T08:38:44.629 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2021-11-19T08:38:44.630 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2021-11-19T08:38:44.630 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120606 s, 869 MB/s 2021-11-19T08:38:44.631 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2021-11-19T08:38:44.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:44 smithi023 conmon[33490]: cluster 2021-11-19T08:38:42.807183+0000 mgr.smithi023.ehlpcl 2021-11-19T08:38:44.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:44 smithi023 conmon[33490]: (mgr.14164) 119 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:44.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:44 smithi023 conmon[33490]: audit 2021-11-19T08:38:43.229599+0000 mon.smithi023 (mon.0) 332 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-11-19T08:38:44.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:44 smithi023 conmon[33490]: cluster 2021-11-19T08:38:43.229646+0000 mon.smithi023 (mon.0) 333 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2021-11-19T08:38:44.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:44 smithi023 conmon[33490]: audit 2021-11-19T08:38:43.229738+0000 mon.smithi023 (mon.0) 334 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:44.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:44 smithi023 conmon[33490]: audit 2021-11-19T08:38:43.230497+0000 mon.smithi023 (mon.0) 335 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-11-19T08:38:45.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi134:vg_nvme/lv_3 2021-11-19T08:38:45.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:45 smithi134 conmon[36363]: audit 2021-11-19T08:38:44.238727+0000 mon.smithi023 (mon.0) 336 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-11-19T08:38:45.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:45 smithi134 conmon[36363]: cluster 2021-11-19T08:38:44.238801+0000 mon.smithi023 (mon.0) 337 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-11-19T08:38:45.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:45 smithi134 conmon[36363]: audit 2021-11-19T08:38:44.238954+0000 mon.smithi023 (mon.0) 338 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:45.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:45 smithi134 conmon[36363]: audit 2021-11-19T08:38:44.946210+0000 mon.smithi023 (mon.0) 339 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:45.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:45 smithi023 conmon[33490]: audit 2021-11-19T08:38:44.238727+0000 2021-11-19T08:38:45.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:45 smithi023 conmon[33490]: mon.smithi023 (mon.0) 336 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-11-19T08:38:45.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:45 smithi023 conmon[33490]: cluster 2021-11-19T08:38:44.238801 2021-11-19T08:38:45.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:45 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 337 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-11-19T08:38:45.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:45 smithi023 conmon[33490]: audit 2021 2021-11-19T08:38:45.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:45 smithi023 conmon[33490]: -11-19T08:38:44. 2021-11-19T08:38:45.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:45 smithi023 conmon[33490]: 238954+0000 mon.smithi023 (mon.0 2021-11-19T08:38:45.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:45 smithi023 conmon[33490]: ) 338 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:45.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:45 smithi023 conmon[33490]: audit 2021 2021-11-19T08:38:45.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:45 smithi023 conmon[33490]: -11-19T08:38:44 2021-11-19T08:38:45.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:45 smithi023 conmon[33490]: .946210+0000 mon.smithi023 (mon. 2021-11-19T08:38:45.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:45 smithi023 conmon[33490]: 0) 339 : audit [INF] 2021-11-19T08:38:45.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:45 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:46.268 INFO:journalctl@ceph.osd.3.smithi134.stdout:Nov 19 08:38:46 smithi134 conmon[42726]: debug 2021-11-19T08:38:46.134+0000 7eff60732080 -1 osd.3 0 log_to_monitors {default=true} 2021-11-19T08:38:46.268 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:46 smithi134 conmon[36363]: cluster 2021-11-19T08:38:44.807537+0000 mgr.smithi023.ehlpcl 2021-11-19T08:38:46.269 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:46 smithi134 conmon[36363]: (mgr.14164) 120 : cluster [DBG] pgmap v84: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:46.269 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:46 smithi134 conmon[36363]: cluster 2021-11-19T08:38:45.949565+0000 mon.smithi023 (mon.0) 340 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-11-19T08:38:46.269 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:46 smithi134 conmon[36363]: audit 2021-11-19T08:38:45.949684+0000 mon.smithi023 (mon.0) 341 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:46.269 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:46 smithi134 conmon[36363]: audit 2021-11-19T08:38:46.138744+0000 mon.smithi023 (mon.0) 342 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-19T08:38:46.270 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:46 smithi134 conmon[36363]: audit 2021-11-19T08:38:46.138759+0000 mon.smithi134 (mon.1) 4 : audit [INF] from='osd.3 [v2:172.21.15.134:6800/2145743581,v1:172.21.15.134:6801/2145743581]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-19T08:38:46.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: cluster 2021-11-19T08:38:44 2021-11-19T08:38:46.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: .807537+0000 mgr.smithi023.ehlpcl (mgr.14164) 120 : cluster [DBG] pgmap v84: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:46.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: cluster 2021-11-19T08:38:45.949565+0000 mon.smithi023 2021-11-19T08:38:46.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: (mon.0) 340 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-11-19T08:38:46.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:38:46.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: 11-19T08:38:45.949684+0000 mon.smithi023 2021-11-19T08:38:46.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: (mon.0) 341 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:46.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: 2021-11-19T08:38:46.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: audit 2021-11-19T08:38 2021-11-19T08:38:46.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: :46.138744+0000 mon.smithi023 (mon.0) 2021-11-19T08:38:46.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: 342 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-19T08:38:46.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: audit 2021-11- 2021-11-19T08:38:46.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: 19T08:38:46.138759+0000 mon.smithi134 2021-11-19T08:38:46.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: (mon.1) 4 : audit [INF] from='osd.3 [v2:172.21.15.134:6800/2145743581,v1:172.21.15.134:6801/2145743581]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-11-19T08:38:46.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:46 smithi023 conmon[33490]: 2021-11-19T08:38:47.169 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.181+0000 7f0fa229b700 1 -- 172.21.15.134:0/2256087030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f9c0f77d0 msgr2=0x7f0f9c0f7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:38:47.169 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.181+0000 7f0fa229b700 1 --2- 172.21.15.134:0/2256087030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f9c0f77d0 0x7f0f9c0f7c30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f0f8c004660 tx=0x7f0f8c009b30).stop 2021-11-19T08:38:47.169 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.181+0000 7f0fa229b700 1 -- 172.21.15.134:0/2256087030 shutdown_connections 2021-11-19T08:38:47.170 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.181+0000 7f0fa229b700 1 --2- 172.21.15.134:0/2256087030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f9c0f77d0 0x7f0f9c0f7c30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:47.170 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.181+0000 7f0fa229b700 1 --2- 172.21.15.134:0/2256087030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f9c0f6530 0x7f0f9c0f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:47.170 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.181+0000 7f0fa229b700 1 -- 172.21.15.134:0/2256087030 >> 172.21.15.134:0/2256087030 conn(0x7f0f9c0f1cd0 msgr2=0x7f0f9c0f40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:38:47.170 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.181+0000 7f0fa229b700 1 -- 172.21.15.134:0/2256087030 shutdown_connections 2021-11-19T08:38:47.171 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.181+0000 7f0fa229b700 1 -- 172.21.15.134:0/2256087030 wait complete. 2021-11-19T08:38:47.171 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.182+0000 7f0fa229b700 1 Processor -- start 2021-11-19T08:38:47.171 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.182+0000 7f0fa229b700 1 -- start start 2021-11-19T08:38:47.171 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.182+0000 7f0fa229b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f9c0f6530 0x7f0f9c106d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:38:47.172 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.182+0000 7f0fa229b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f9c0f77d0 0x7f0f9c107230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:38:47.172 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.182+0000 7f0fa229b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c107830 con 0x7f0f9c0f77d0 2021-11-19T08:38:47.172 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.182+0000 7f0fa229b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c108040 con 0x7f0f9c0f6530 2021-11-19T08:38:47.172 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.182+0000 7f0fa1299700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f9c0f6530 0x7f0f9c106d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:38:47.173 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.182+0000 7f0fa1299700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f9c0f6530 0x7f0f9c106d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47814/0 (socket says 172.21.15.134:47814) 2021-11-19T08:38:47.173 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.182+0000 7f0fa1299700 1 -- 172.21.15.134:0/3489040498 learned_addr learned my addr 172.21.15.134:0/3489040498 (peer_addr_for_me v2:172.21.15.134:0/0) 2021-11-19T08:38:47.173 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.182+0000 7f0fa0a98700 1 --2- 172.21.15.134:0/3489040498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f9c0f77d0 0x7f0f9c107230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:38:47.173 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.182+0000 7f0fa1299700 1 -- 172.21.15.134:0/3489040498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f9c0f77d0 msgr2=0x7f0f9c107230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:38:47.174 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.182+0000 7f0fa1299700 1 --2- 172.21.15.134:0/3489040498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f9c0f77d0 0x7f0f9c107230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:47.174 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.182+0000 7f0fa1299700 1 -- 172.21.15.134:0/3489040498 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c005040 con 0x7f0f9c0f6530 2021-11-19T08:38:47.174 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.183+0000 7f0fa1299700 1 --2- 172.21.15.134:0/3489040498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f9c0f6530 0x7f0f9c106d40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0f9800e7c0 tx=0x7f0f9800b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:38:47.174 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.183+0000 7f0f91ffb700 1 -- 172.21.15.134:0/3489040498 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f980092e0 con 0x7f0f9c0f6530 2021-11-19T08:38:47.175 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.183+0000 7f0f91ffb700 1 -- 172.21.15.134:0/3489040498 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f9800f040 con 0x7f0f9c0f6530 2021-11-19T08:38:47.175 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.183+0000 7f0fa229b700 1 -- 172.21.15.134:0/3489040498 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c1082c0 con 0x7f0f9c0f6530 2021-11-19T08:38:47.175 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.183+0000 7f0f91ffb700 1 -- 172.21.15.134:0/3489040498 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f980147c0 con 0x7f0f9c0f6530 2021-11-19T08:38:47.175 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.183+0000 7f0fa229b700 1 -- 172.21.15.134:0/3489040498 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c108880 con 0x7f0f9c0f6530 2021-11-19T08:38:47.177 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.184+0000 7f0f91ffb700 1 -- 172.21.15.134:0/3489040498 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0f9801b030 con 0x7f0f9c0f6530 2021-11-19T08:38:47.177 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.184+0000 7f0fa229b700 1 -- 172.21.15.134:0/3489040498 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c059100 con 0x7f0f9c0f6530 2021-11-19T08:38:47.177 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.184+0000 7f0f91ffb700 1 --2- 172.21.15.134:0/3489040498 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f8805c300 0x7f0f8805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:38:47.177 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.185+0000 7f0f91ffb700 1 -- 172.21.15.134:0/3489040498 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 3282+0+0 (secure 0 0 0) 0x7f0f980755a0 con 0x7f0f9c0f6530 2021-11-19T08:38:47.178 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.185+0000 7f0fa0a98700 1 --2- 172.21.15.134:0/3489040498 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f8805c300 0x7f0f8805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:38:47.178 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.185+0000 7f0fa0a98700 1 --2- 172.21.15.134:0/3489040498 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f8805c300 0x7f0f8805e7c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f0f8c005010 tx=0x7f0f8c016040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:38:47.178 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.188+0000 7f0f91ffb700 1 -- 172.21.15.134:0/3489040498 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f98048a90 con 0x7f0f9c0f6530 2021-11-19T08:38:47.360 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:47.374+0000 7f0fa229b700 1 -- 172.21.15.134:0/3489040498 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f9c0008d0 con 0x7f0f8805c300 2021-11-19T08:38:47.948 INFO:journalctl@ceph.osd.3.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[42726]: debug 2021-11-19T08:38:47.956+0000 7eff56ca3700 -1 osd.3 0 waiting for initial osdmap 2021-11-19T08:38:47.949 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[36363]: cluster 2021-11-19T08:38:46.807802+0000 2021-11-19T08:38:47.949 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[36363]: mgr.smithi023.ehlpcl (mgr.14164) 121 : cluster [DBG] pgmap v86: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:47.949 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[36363]: audit 2021-11-19T08:38:46.951225+0000 mon.smithi023 (mon.0) 343 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-11-19T08:38:47.949 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[36363]: cluster 2021-11-19T08:38:46.951318+0000 mon.smithi023 (mon.0) 344 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-11-19T08:38:47.950 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[36363]: audit 2021-11-19T08:38:46.951440+0000 mon.smithi023 (mon.0) 345 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:47.950 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[36363]: audit 2021-11-19T08:38:46.956058+0000 mon.smithi023 (mon.0) 346 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2021-11-19T08:38:47.950 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[36363]: audit 2021-11-19T08:38:46.956108+0000 mon.smithi134 (mon.1) 5 : audit [INF] from='osd.3 [v2:172.21.15.134:6800/2145743581,v1:172.21.15.134:6801/2145743581]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2021-11-19T08:38:47.950 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[36363]: audit 2021-11-19T08:38:46.966142+0000 mon.smithi023 (mon.0) 347 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:47.951 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[36363]: audit 2021-11-19T08:38:47.376468+0000 mgr.smithi023.ehlpcl (mgr.14164) 122 : audit [DBG] from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:38:47.951 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[36363]: audit 2021-11-19T08:38:47.377246+0000 mon.smithi023 (mon.0) 348 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-19T08:38:47.951 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[36363]: cephadm 2021-11-19T08:38:47.378436+0000 mgr.smithi023.ehlpcl (mgr.14164) 123 : cephadm [INF] Applying service osd.None on host smithi134... 2021-11-19T08:38:47.952 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[36363]: audit 2021-11-19T08:38:47.378672+0000 mon.smithi023 (mon.0) 349 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-19T08:38:47.952 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[36363]: audit 2021-11-19T08:38:47.379335+0000 mon.smithi023 (mon.0) 350 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:48.290 INFO:journalctl@ceph.osd.3.smithi134.stdout:Nov 19 08:38:47 smithi134 conmon[42726]: debug 2021-11-19T08:38:47.963+0000 7eff4fc18700 -1 osd.3 25 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-19T08:38:48.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: cluster 2021-11-19T08:38:46.807802+0000 2021-11-19T08:38:48.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 121 : cluster [DBG] pgmap v86: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:48.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: audit 2021-11-19T08:38:46 2021-11-19T08:38:48.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: .951225+0000 mon.smithi023 (mon.0) 343 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-11-19T08:38:48.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: cluster 2021-11-19T08 2021-11-19T08:38:48.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: :38:46.951318+0000 mon.smithi023 (mon.0) 344 2021-11-19T08:38:48.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-11-19T08:38:48.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: audit 2021-11-19T08:38:48.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: 2021-11-19T08:38:46.951440+0000 2021-11-19T08:38:48.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: mon.smithi023 (mon.0) 345 : audit [DBG] 2021-11-19T08:38:48.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:48.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:38:48.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: -19T08:38:46.956058+0000 mon.smithi023 (mon.0 2021-11-19T08:38:48.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: ) 346 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2021-11-19T08:38:48.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:38:48.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: -19T08:38:46.956108+0000 mon.smithi134 (mon 2021-11-19T08:38:48.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: .1) 5 : audit [INF] 2021-11-19T08:38:48.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: from='osd.3 [v2:172.21.15.134:6800/2145743581,v1:172.21.15.134:6801/2145743581]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2021-11-19T08:38:48.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: audit 2021-11-19T08:38:48.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: 2021-11-19T08:38:46.966142+0000 2021-11-19T08:38:48.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: mon.smithi023 (mon.0) 347 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:48.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: 2021-11-19T08:38:48.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: audit 2021-11-19T08:38:48.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: 2021-11-19T08:38:47.376468+0000 2021-11-19T08:38:48.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 122 : audit [DBG] from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:38:48.398 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: audit 2021-11- 2021-11-19T08:38:48.398 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: 19T08:38:47.377246+0000 mon.smithi023 (mon.0) 2021-11-19T08:38:48.398 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: 348 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-19T08:38:48.398 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: cephadm 2021-11-19T08:38:47.378436+0000 mgr.smithi023.ehlpcl (mgr.14164) 123 : cephadm [INF] Applying service osd.None on host smithi134... 2021-11-19T08:38:48.399 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: audit 2021-11-19T08:38:47.378672+0000 mon.smithi023 (mon.0) 349 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-19T08:38:48.399 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:47 smithi023 conmon[33490]: audit 2021-11-19T08:38:47.379335+0000 mon.smithi023 (mon.0) 350 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:49.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:48 smithi134 conmon[36363]: audit 2021-11-19T08:38:47.954317+0000 mon.smithi023 (mon.0) 351 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2021-11-19T08:38:49.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:48 smithi134 conmon[36363]: cluster 2021-11-19T08:38:47.954438+0000 mon.smithi023 (mon.0) 352 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2021-11-19T08:38:49.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:48 smithi134 conmon[36363]: audit 2021-11-19T08:38:47.955236+0000 mon.smithi023 (mon.0) 353 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:49.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:48 smithi134 conmon[36363]: audit 2021-11-19T08:38:47.960925+0000 mon.smithi023 (mon.0) 354 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:49.292 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:48 smithi134 conmon[36363]: cephadm 2021-11-19T08:38:48.077867+0000 mgr.smithi023.ehlpcl (mgr.14164) 124 : cephadm [INF] Detected new or changed devices on smithi134 2021-11-19T08:38:49.292 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:48 smithi134 conmon[36363]: audit 2021-11-19T08:38:48.082642+0000 mon.smithi023 (mon.0) 355 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:49.292 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:48 smithi134 conmon[36363]: audit 2021-11-19T08:38:48.088134+0000 mon.smithi023 (mon.0) 356 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:49.292 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:48 smithi134 conmon[36363]: audit 2021-11-19T08:38:48.091704+0000 mon.smithi023 (mon.0) 357 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:38:49.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: audit 2021-11-19T08:38:47. 2021-11-19T08:38:49.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: 954317+0000 mon.smithi023 (mon.0) 351 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2021-11-19T08:38:49.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: cluster 2021-11-19T08:38:47. 2021-11-19T08:38:49.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: 954438+0000 mon.smithi023 (mon.0) 352 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2021-11-19T08:38:49.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: audit 2021-11-19T08:38: 2021-11-19T08:38:49.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: 47.955236+0000 mon.smithi023 (mon.0) 353 2021-11-19T08:38:49.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:49.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: audit 2021-11-19T08 2021-11-19T08:38:49.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: :38:47.960925+0000 mon.smithi023 (mon.0) 354 : 2021-11-19T08:38:49.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:49.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: cephadm 2021-11-19T 2021-11-19T08:38:49.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: 08:38:48.077867+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:38:49.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: 14164) 124 : cephadm [INF] Detected new or changed devices on smithi134 2021-11-19T08:38:49.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: audit 2021-11- 2021-11-19T08:38:49.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: 19T08:38:48.082642+0000 mon.smithi023 (mon 2021-11-19T08:38:49.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: .0) 355 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:49.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:38:49.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: T08:38:48.088134+0000 mon.smithi023 ( 2021-11-19T08:38:49.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: mon.0) 356 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:49.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: audit 2021 2021-11-19T08:38:49.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: -11-19T08:38:48.091704+0000 2021-11-19T08:38:49.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: mon.smithi023 (mon.0) 357 : audit 2021-11-19T08:38:49.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:48 smithi023 conmon[33490]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:38:50.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: cluster 2021-11-19T08:38:47.180887+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-19T08:38:50.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: cluster 2021-11-19T08:38:47.181000+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-19T08:38:50.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: cluster 2021-11-19T08:38:48.808088+0000 mgr.smithi023.ehlpcl (mgr.14164) 125 : cluster [DBG] pgmap v89: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:50.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: audit 2021-11-19T08:38:48.958072+0000 mon.smithi023 (mon.0) 358 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:50.292 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: cluster 2021- 2021-11-19T08:38:50.292 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: 11-19T08:38:48.961705+0000 mon.smithi023 (mon.0) 359 : cluster [INF] osd.3 [v2:172.21.15.134:6800/2145743581,v1:172.21.15.134:6801/2145743581] boot 2021-11-19T08:38:50.292 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: cluster 2021-11-19T08:38: 2021-11-19T08:38:50.292 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: 48.961737+0000 mon.smithi023 (mon.0) 360 : 2021-11-19T08:38:50.293 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-11-19T08:38:50.293 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: audit 2021-11-19T08:38:50.293 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: 2021-11-19T08:38:48 2021-11-19T08:38:50.293 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: .962180+0000 mon.smithi023 (mon.0 2021-11-19T08:38:50.294 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: ) 361 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:50.294 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: cluster 2021-11-19T08:38:49.149362+0000 mon.smithi023 (mon.0) 362 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2021-11-19T08:38:50.294 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: audit 2021-11-19T08:38:50.294 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: 2021-11-19 2021-11-19T08:38:50.295 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: T08:38:49.210529+0000 mon.smithi134 2021-11-19T08:38:50.295 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: (mon.1) 6 : audit 2021-11-19T08:38:50.295 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: [INF] from='client.? 172.21.15.134:0/2387984634' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "015b6da1-35ed-43f4-84c9-337d2b0e79bb"}]: dispatch 2021-11-19T08:38:50.295 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: audit 2021-11 2021-11-19T08:38:50.295 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: -19T08:38:49.210532 2021-11-19T08:38:50.296 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 363 : 2021-11-19T08:38:50.296 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "015b6da1-35ed-43f4-84c9-337d2b0e79bb"}]: dispatch 2021-11-19T08:38:50.296 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: audit 2021 2021-11-19T08:38:50.296 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: -11-19T08:38:49. 2021-11-19T08:38:50.296 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: 214538+0000 mon.smithi023 (mon.0 2021-11-19T08:38:50.297 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: ) 364 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "015b6da1-35ed-43f4-84c9-337d2b0e79bb"}]': finished 2021-11-19T08:38:50.297 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: cluster 2021 2021-11-19T08:38:50.297 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: -11-19T08:38:49 2021-11-19T08:38:50.297 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: .214601+0000 mon.smithi023 (mon.0 2021-11-19T08:38:50.298 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: ) 365 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-11-19T08:38:50.298 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: audit 2021 2021-11-19T08:38:50.298 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: -11-19T08:38:49 2021-11-19T08:38:50.298 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: .214712+0000 mon.smithi023 (mon.0 2021-11-19T08:38:50.299 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: ) 366 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-19T08:38:50.299 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: audit 2021 2021-11-19T08:38:50.299 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: -11-19T08:38:49 2021-11-19T08:38:50.299 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: .631196+0000 mon.smithi134 (mon. 2021-11-19T08:38:50.299 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: 1) 7 : audit [DBG] from='client.? 172.21.15.134:0/210405855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-19T08:38:50.300 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:50 smithi134 conmon[36363]: 2021-11-19T08:38:50.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: cluster 2021-11-19T08:38:47.180887+0000 osd.3 (osd. 2021-11-19T08:38:50.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 3) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-19T08:38:50.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: cluster 2021-11-19T08:38:47.181000+0000 osd.3 (osd.3 2021-11-19T08:38:50.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: ) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-19T08:38:50.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: cluster 2021-11-19T08:38:50.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 2021-11-19T08:38:48.808088+0000 2021-11-19T08:38:50.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 125 : cluster [DBG] 2021-11-19T08:38:50.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: pgmap v89: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-11-19T08:38:50.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: audit 2021-11-19T08:38:50.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 2021-11-19T08:38:48.958072+0000 2021-11-19T08:38:50.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: mon.smithi023 (mon.0) 358 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:50.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 2021-11-19T08:38:50.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: cluster 2021-11-19T08:38:50.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 2021-11-19T08:38:48.961705+0000 mon.smithi023 (mon.0) 359 : cluster [INF] 2021-11-19T08:38:50.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: osd.3 [v2:172.21.15.134:6800/2145743581,v1:172.21.15.134:6801/2145743581] boot 2021-11-19T08:38:50.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: cluster 2021-11- 2021-11-19T08:38:50.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 19T08:38:48.961737+0000 mon.smithi023 (mon.0) 2021-11-19T08:38:50.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 360 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-11-19T08:38:50.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: audit 2021 2021-11-19T08:38:50.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: -11-19T08:38:48.962180 2021-11-19T08:38:50.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 361 : audit 2021-11-19T08:38:50.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-11-19T08:38:50.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: cluster 2021-11-19T08:38 2021-11-19T08:38:50.398 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: :49.149362+0000 mon.smithi023 (mon.0) 2021-11-19T08:38:50.398 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 362 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2021-11-19T08:38:50.398 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: audit 2021-11- 2021-11-19T08:38:50.398 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 19T08:38:49.210529+0000 mon.smithi134 2021-11-19T08:38:50.399 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: (mon.1) 6 : audit [INF] from='client.? 172.21.15.134:0/2387984634' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "015b6da1-35ed-43f4-84c9-337d2b0e79bb"}]: dispatch 2021-11-19T08:38:50.399 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 2021-11-19T08:38:50.399 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: audit 2021-11-19T08:38:50.399 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 2021-11-19T08:38:49.210532+0000 2021-11-19T08:38:50.400 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: mon.smithi023 (mon.0) 363 : audit [INF] 2021-11-19T08:38:50.400 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "015b6da1-35ed-43f4-84c9-337d2b0e79bb"}]: dispatch 2021-11-19T08:38:50.400 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: audit 2021-11-19T08:38:50.400 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 2021-11-19T08:38:49.214538+0000 mon.smithi023 (mon.0) 2021-11-19T08:38:50.400 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 364 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "015b6da1-35ed-43f4-84c9-337d2b0e79bb"}]': finished 2021-11-19T08:38:50.401 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: cluster 2021-11-19T08:38: 2021-11-19T08:38:50.401 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 49.214601+0000 mon.smithi023 (mon.0) 365 2021-11-19T08:38:50.401 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-11-19T08:38:50.401 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: audit 2021-11-19T 2021-11-19T08:38:50.402 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 08:38:49.214712+0000 mon.smithi023 (mon 2021-11-19T08:38:50.402 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: .0) 366 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-19T08:38:50.402 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:38:50.402 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: 11-19T08:38:49.631196 2021-11-19T08:38:50.403 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: +0000 mon.smithi134 (mon.1) 7 : audit 2021-11-19T08:38:50.403 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:49 smithi023 conmon[33490]: [DBG] from='client.? 172.21.15.134:0/210405855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-19T08:38:52.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:51 smithi134 conmon[36363]: cluster 2021-11-19T08:38:50.808569+0000 mgr.smithi023.ehlpcl (mgr.14164) 126 2021-11-19T08:38:52.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:51 smithi134 conmon[36363]: : cluster [DBG] pgmap v93: 1 pgs: 1 activating; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-19T08:38:52.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:51 smithi023 conmon[33490]: cluster 2021-11-19T08:38:50.808569+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:38:52.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:51 smithi023 conmon[33490]: mgr.14164) 126 : cluster [DBG] pgmap v93: 1 pgs: 1 activating; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-19T08:38:54.201 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:54 smithi134 conmon[36363]: cluster 2021-11-19T08:38:52.809020+0000 2021-11-19T08:38:54.202 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:54 smithi134 conmon[36363]: mgr.smithi023.ehlpcl (mgr.14164) 127 : cluster [DBG] pgmap v94: 1 pgs: 1 activating; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-19T08:38:54.202 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:54 smithi134 conmon[36363]: audit 2021-11-19T08:38:53.023768+0000 mon.smithi023 (mon.0) 367 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:54.202 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:54 smithi134 conmon[36363]: audit 2021-11-19T08:38:53.026881+0000 mon.smithi023 (mon.0) 368 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:54.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:54 smithi023 conmon[33490]: cluster 2021-11-19T08:38:52.809020+0000 mgr.smithi023.ehlpcl (mgr.14164) 127 2021-11-19T08:38:54.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:54 smithi023 conmon[33490]: : cluster [DBG] pgmap v94: 1 pgs: 1 activating; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-19T08:38:54.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:54 smithi023 conmon[33490]: audit 2021-11-19T08:38:53.023768+0000 mon.smithi023 (mon.0) 367 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:54.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:54 smithi023 conmon[33490]: audit 2021-11-19T08:38:53.026881 2021-11-19T08:38:54.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:54 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 368 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:56.372 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:56 smithi134 conmon[36363]: cluster 2021-11-19T08:38:54.809585+0000 mgr.smithi023.ehlpcl (mgr.14164) 128 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-19T08:38:56.372 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:56 smithi134 conmon[36363]: audit 2021-11-19T08:38:55.284593+0000 mon.smithi023 (mon.0) 369 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-11-19T08:38:56.373 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:56 smithi134 conmon[36363]: audit 2021-11-19T08:38:55.285546+0000 mon.smithi023 (mon.0) 370 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:56.373 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:56 smithi134 conmon[36363]: cephadm 2021-11-19T08:38:55.286339+0000 mgr.smithi023.ehlpcl (mgr.14164) 129 : cephadm [INF] Deploying daemon osd.4 on smithi134 2021-11-19T08:38:56.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:56 smithi023 conmon[33490]: cluster 2021-11-19T08:38:54.809585+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:38:56.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:56 smithi023 conmon[33490]: mgr.14164) 128 : cluster [DBG] pgmap v95: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-19T08:38:56.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:56 smithi023 conmon[33490]: audit 2021-11-19T08:38:55.284593 2021-11-19T08:38:56.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:56 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 369 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-11-19T08:38:56.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:56 smithi023 conmon[33490]: audit 2021-11- 2021-11-19T08:38:56.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:56 smithi023 conmon[33490]: 19T08:38:55.285546+0000 mon.smithi023 (mon.0) 2021-11-19T08:38:56.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:56 smithi023 conmon[33490]: 370 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:38:56.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:56 smithi023 conmon[33490]: cephadm 2021-11-19T 2021-11-19T08:38:56.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:56 smithi023 conmon[33490]: 08:38:55.286339+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:38:56.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:56 smithi023 conmon[33490]: .14164) 129 : cephadm [INF] Deploying daemon osd.4 on smithi134 2021-11-19T08:38:58.084 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 4 on host 'smithi134' 2021-11-19T08:38:58.085 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:58.097+0000 7f0f91ffb700 1 -- 172.21.15.134:0/3489040498 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0f9c0008d0 con 0x7f0f8805c300 2021-11-19T08:38:58.087 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:58.100+0000 7f0fa229b700 1 -- 172.21.15.134:0/3489040498 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f8805c300 msgr2=0x7f0f8805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:38:58.087 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:58.100+0000 7f0fa229b700 1 --2- 172.21.15.134:0/3489040498 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f8805c300 0x7f0f8805e7c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f0f8c005010 tx=0x7f0f8c016040).stop 2021-11-19T08:38:58.088 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:58.100+0000 7f0fa229b700 1 -- 172.21.15.134:0/3489040498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f9c0f6530 msgr2=0x7f0f9c106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:38:58.088 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:58.100+0000 7f0fa229b700 1 --2- 172.21.15.134:0/3489040498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f9c0f6530 0x7f0f9c106d40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0f9800e7c0 tx=0x7f0f9800b040).stop 2021-11-19T08:38:58.089 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:58.100+0000 7f0fa229b700 1 -- 172.21.15.134:0/3489040498 shutdown_connections 2021-11-19T08:38:58.089 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:58.100+0000 7f0fa229b700 1 --2- 172.21.15.134:0/3489040498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f9c0f6530 0x7f0f9c106d40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:58.089 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:58.100+0000 7f0fa229b700 1 --2- 172.21.15.134:0/3489040498 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f8805c300 0x7f0f8805e7c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:58.090 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:58.100+0000 7f0fa229b700 1 --2- 172.21.15.134:0/3489040498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f9c0f77d0 0x7f0f9c107230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:38:58.090 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:58.100+0000 7f0fa229b700 1 -- 172.21.15.134:0/3489040498 >> 172.21.15.134:0/3489040498 conn(0x7f0f9c0f1cd0 msgr2=0x7f0f9c0fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:38:58.090 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:58.101+0000 7f0fa229b700 1 -- 172.21.15.134:0/3489040498 shutdown_connections 2021-11-19T08:38:58.091 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:38:58.101+0000 7f0fa229b700 1 -- 172.21.15.134:0/3489040498 wait complete. 2021-11-19T08:38:58.411 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:58 smithi134 conmon[36363]: cluster 2021-11-19T08:38:56.810034+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:38:58.411 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:58 smithi134 conmon[36363]: .14164) 130 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-19T08:38:58.412 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:58 smithi134 conmon[36363]: audit 2021-11-19T08:38:58.082432+0000 mon.smithi023 (mon.0) 371 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:38:58.412 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:58 smithi134 conmon[36363]: audit 2021-11-19T08:38:58.090510+0000 mon.smithi023 (mon.0) 372 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:58.412 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:38:58 smithi134 conmon[36363]: audit 2021-11-19T08:38:58.096912+0000 mon.smithi023 (mon.0) 373 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:58.445 DEBUG:teuthology.orchestra.run.smithi134:osd.4> sudo journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.4.service 2021-11-19T08:38:58.448 INFO:tasks.cephadm:Deploying osd.5 on smithi134 with /dev/vg_nvme/lv_2... 2021-11-19T08:38:58.449 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-11-19T08:38:58.476 INFO:journalctl@ceph.osd.4.smithi134.stdout:-- Logs begin at Fri 2021-11-19 08:24:18 UTC. -- 2021-11-19T08:38:58.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:58 smithi023 conmon[33490]: cluster 2021-11-19T08:38:56. 2021-11-19T08:38:58.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:58 smithi023 conmon[33490]: 810034+0000 mgr.smithi023.ehlpcl (mgr.14164) 130 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-19T08:38:58.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:58 smithi023 conmon[33490]: audit 2021-11-19T08:38:58.082432+0000 mon.smithi023 (mon.0) 371 2021-11-19T08:38:58.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:58 smithi023 conmon[33490]: : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:38:58.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:58 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:38:58.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:58 smithi023 conmon[33490]: 11-19T08:38:58.090510+0000 mon.smithi023 (mon. 2021-11-19T08:38:58.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:58 smithi023 conmon[33490]: 0) 372 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:58.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:58 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:38:58.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:58 smithi023 conmon[33490]: T08:38:58.096912+0000 mon.smithi023 ( 2021-11-19T08:38:58.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:38:58 smithi023 conmon[33490]: mon.0) 373 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:38:58.790 INFO:journalctl@ceph.osd.4.smithi134.stdout:Nov 19 08:38:58 smithi134 conmon[47596]: debug 2021-11-19T08:38:58.550+0000 7f59077e4080 -1 Falling back to public interface 2021-11-19T08:39:00.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:00 smithi134 conmon[36363]: cluster 2021-11-19T08:38:58.810319+0000 mgr.smithi023.ehlpcl (mgr.14164) 131 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-19T08:39:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-11-19T08:39:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-11-19T08:39:00.473 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2021-11-19T08:39:00.473 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2021-11-19T08:39:00.473 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121593 s, 862 MB/s 2021-11-19T08:39:00.473 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2021-11-19T08:39:00.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:00 smithi023 conmon[33490]: cluster 2021-11-19T08:38:58.810319 2021-11-19T08:39:00.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:00 smithi023 conmon[33490]: +0000 mgr.smithi023.ehlpcl (mgr.14164) 131 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-19T08:39:00.903 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd smithi134:vg_nvme/lv_2 2021-11-19T08:39:02.220 INFO:journalctl@ceph.osd.4.smithi134.stdout:Nov 19 08:39:02 smithi134 conmon[47596]: debug 2021-11-19T08:39:02.081+0000 7f59077e4080 -1 osd.4 0 log_to_monitors {default=true} 2021-11-19T08:39:02.220 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:02 smithi134 conmon[36363]: cluster 2021-11-19T08:39:00.810631+0000 2021-11-19T08:39:02.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:02 smithi134 conmon[36363]: mgr.smithi023.ehlpcl (mgr.14164) 132 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-19T08:39:02.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:02 smithi134 conmon[36363]: audit 2021-11-19T08:39:01.121299+0000 mon.smithi023 (mon.0) 374 : audit [INF] 2021-11-19T08:39:02.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:02 smithi134 conmon[36363]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:02.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:02 smithi134 conmon[36363]: audit 2021-11- 2021-11-19T08:39:02.222 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:02 smithi134 conmon[36363]: 19T08:39:02.084871+0000 mon.smithi023 (mon.0 2021-11-19T08:39:02.222 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:02 smithi134 conmon[36363]: ) 375 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-19T08:39:02.222 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:02 smithi134 conmon[36363]: audit 2021 2021-11-19T08:39:02.223 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:02 smithi134 conmon[36363]: -11-19T08:39:02 2021-11-19T08:39:02.223 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:02 smithi134 conmon[36363]: .084989+0000 mon.smithi134 (mon.1) 8 2021-11-19T08:39:02.223 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:02 smithi134 conmon[36363]: : audit [INF] from='osd.4 [v2:172.21.15.134:6808/1867765886,v1:172.21.15.134:6809/1867765886]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-19T08:39:02.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:02 smithi023 conmon[33490]: cluster 2021-11-19T08:39:00.810631+0000 mgr.smithi023.ehlpcl (mgr.14164) 132 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-19T08:39:02.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:02 smithi023 conmon[33490]: audit 2021-11-19T08:39:01.121299+0000 mon.smithi023 (mon.0) 374 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:02.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:02 smithi023 conmon[33490]: audit 2021-11-19T08:39:02.084871+0000 mon.smithi023 (mon.0) 375 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-19T08:39:02.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:02 smithi023 conmon[33490]: audit 2021-11-19T08:39:02.084989+0000 mon.smithi134 (mon.1) 8 : audit [INF] from='osd.4 [v2:172.21.15.134:6808/1867765886,v1:172.21.15.134:6809/1867765886]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-11-19T08:39:02.524 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.537+0000 7f3795083700 1 -- 172.21.15.134:0/1599858778 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37900f66e0 msgr2=0x7f37900f6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:02.525 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.537+0000 7f3795083700 1 --2- 172.21.15.134:0/1599858778 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37900f66e0 0x7f37900f6b00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f3780004660 tx=0x7f3780009b30).stop 2021-11-19T08:39:02.525 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.537+0000 7f3795083700 1 -- 172.21.15.134:0/1599858778 shutdown_connections 2021-11-19T08:39:02.525 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.537+0000 7f3795083700 1 --2- 172.21.15.134:0/1599858778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37900f6ff0 0x7f37900f7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:02.525 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.537+0000 7f3795083700 1 --2- 172.21.15.134:0/1599858778 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37900f66e0 0x7f37900f6b00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:02.526 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.537+0000 7f3795083700 1 -- 172.21.15.134:0/1599858778 >> 172.21.15.134:0/1599858778 conn(0x7f37900f1f00 msgr2=0x7f37900f4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:02.526 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.537+0000 7f3795083700 1 -- 172.21.15.134:0/1599858778 shutdown_connections 2021-11-19T08:39:02.526 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.537+0000 7f3795083700 1 -- 172.21.15.134:0/1599858778 wait complete. 2021-11-19T08:39:02.526 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.538+0000 7f3795083700 1 Processor -- start 2021-11-19T08:39:02.526 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.538+0000 7f3795083700 1 -- start start 2021-11-19T08:39:02.527 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.538+0000 7f3795083700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37900f66e0 0x7f379010d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:02.527 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.538+0000 7f3795083700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37900f6ff0 0x7f379010da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:02.527 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.538+0000 7f3795083700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379010dff0 con 0x7f37900f66e0 2021-11-19T08:39:02.527 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.538+0000 7f3795083700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379010e130 con 0x7f37900f6ff0 2021-11-19T08:39:02.528 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.538+0000 7f378e59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37900f6ff0 0x7f379010da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:02.528 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.538+0000 7f378e59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37900f6ff0 0x7f379010da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47878/0 (socket says 172.21.15.134:47878) 2021-11-19T08:39:02.528 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.538+0000 7f378e59c700 1 -- 172.21.15.134:0/2470521393 learned_addr learned my addr 172.21.15.134:0/2470521393 (peer_addr_for_me v2:172.21.15.134:0/0) 2021-11-19T08:39:02.528 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.538+0000 7f378ed9d700 1 --2- 172.21.15.134:0/2470521393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37900f66e0 0x7f379010d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:02.528 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.538+0000 7f378e59c700 1 -- 172.21.15.134:0/2470521393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37900f66e0 msgr2=0x7f379010d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:02.529 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.538+0000 7f378e59c700 1 --2- 172.21.15.134:0/2470521393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37900f66e0 0x7f379010d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:02.529 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.538+0000 7f378e59c700 1 -- 172.21.15.134:0/2470521393 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780005040 con 0x7f37900f6ff0 2021-11-19T08:39:02.529 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.538+0000 7f378e59c700 1 --2- 172.21.15.134:0/2470521393 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37900f6ff0 0x7f379010da40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f378400e7f0 tx=0x7f378400b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:02.529 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.539+0000 7f37777fe700 1 -- 172.21.15.134:0/2470521393 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37840092e0 con 0x7f37900f6ff0 2021-11-19T08:39:02.530 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.539+0000 7f3795083700 1 -- 172.21.15.134:0/2470521393 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379010ead0 con 0x7f37900f6ff0 2021-11-19T08:39:02.530 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.539+0000 7f37777fe700 1 -- 172.21.15.134:0/2470521393 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f378400f040 con 0x7f37900f6ff0 2021-11-19T08:39:02.530 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.539+0000 7f37777fe700 1 -- 172.21.15.134:0/2470521393 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3784014750 con 0x7f37900f6ff0 2021-11-19T08:39:02.530 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.539+0000 7f3795083700 1 -- 172.21.15.134:0/2470521393 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379010f090 con 0x7f37900f6ff0 2021-11-19T08:39:02.532 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.540+0000 7f3795083700 1 -- 172.21.15.134:0/2470521393 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3790045ad0 con 0x7f37900f6ff0 2021-11-19T08:39:02.532 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.540+0000 7f37777fe700 1 -- 172.21.15.134:0/2470521393 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f378401b030 con 0x7f37900f6ff0 2021-11-19T08:39:02.532 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.540+0000 7f37777fe700 1 --2- 172.21.15.134:0/2470521393 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f377805c2b0 0x7f377805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:02.532 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.541+0000 7f37777fe700 1 -- 172.21.15.134:0/2470521393 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3824+0+0 (secure 0 0 0) 0x7f3784075640 con 0x7f37900f6ff0 2021-11-19T08:39:02.533 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.541+0000 7f378ed9d700 1 --2- 172.21.15.134:0/2470521393 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f377805c2b0 0x7f377805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:02.533 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.541+0000 7f378ed9d700 1 --2- 172.21.15.134:0/2470521393 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f377805c2b0 0x7f377805e770 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f3780005010 tx=0x7f3780016040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:02.533 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.545+0000 7f37777fe700 1 -- 172.21.15.134:0/2470521393 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37840489d0 con 0x7f37900f6ff0 2021-11-19T08:39:02.714 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:02.729+0000 7f3795083700 1 -- 172.21.15.134:0/2470521393 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f379006b780 con 0x7f377805c2b0 2021-11-19T08:39:03.217 INFO:journalctl@ceph.osd.4.smithi134.stdout:Nov 19 08:39:03 smithi134 conmon[47596]: debug 2021-11-19T08:39:03.178+0000 7f58fdd55700 -1 osd.4 0 waiting for initial osdmap 2021-11-19T08:39:03.217 INFO:journalctl@ceph.osd.4.smithi134.stdout:Nov 19 08:39:03 smithi134 conmon[47596]: debug 2021-11-19T08:39:03.185+0000 7f58f64c9700 -1 osd.4 30 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-19T08:39:03.218 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:03 smithi134 conmon[36363]: audit 2021-11-19T08:39:02.168341+0000 mon.smithi023 (mon.0) 376 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-11-19T08:39:03.218 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:03 smithi134 conmon[36363]: cluster 2021-11-19T08:39:02.168418+0000 mon.smithi023 (mon.0) 377 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-11-19T08:39:03.218 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:03 smithi134 conmon[36363]: audit 2021-11-19T08:39:02.168536+0000 mon.smithi023 (mon.0) 378 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-19T08:39:03.218 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:03 smithi134 conmon[36363]: audit 2021-11-19T08:39:02.169283+0000 mon.smithi023 (mon.0) 379 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2021-11-19T08:39:03.219 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:03 smithi134 conmon[36363]: audit 2021-11-19T08:39:02.169373+0000 mon.smithi134 (mon.1) 9 : audit [INF] from='osd.4 [v2:172.21.15.134:6808/1867765886,v1:172.21.15.134:6809/1867765886]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2021-11-19T08:39:03.219 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:03 smithi134 conmon[36363]: audit 2021-11-19T08:39:02.730565+0000 mgr.smithi023.ehlpcl (mgr.14164) 133 : audit [DBG] from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:39:03.219 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:03 smithi134 conmon[36363]: audit 2021-11-19T08:39:02.731385+0000 mon.smithi023 (mon.0) 2021-11-19T08:39:03.219 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:03 smithi134 conmon[36363]: 380 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-19T08:39:03.220 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:03 smithi134 conmon[36363]: cephadm 2021-11-19T08:39:02.732562+0000 mgr.smithi023.ehlpcl (mgr.14164) 134 : cephadm [INF] Applying service osd.None on host smithi134... 2021-11-19T08:39:03.220 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:03 smithi134 conmon[36363]: audit 2021-11-19T08:39:02.732791+0000 mon.smithi023 (mon.0) 381 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-19T08:39:03.220 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:03 smithi134 conmon[36363]: audit 2021-11-19T08:39:02.733381+0000 mon.smithi023 (mon.0) 382 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:39:03.221 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:03 smithi134 conmon[36363]: audit 2021-11-19T08:39:02.923367+0000 mon.smithi023 (mon.0) 383 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:03.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: audit 2021-11-19T08:39:02.168341+0000 mon.smithi023 (mon. 2021-11-19T08:39:03.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 0) 376 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-11-19T08:39:03.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: cluster 2021-11-19T08:39:02.168418+0000 mon.smithi023 (mon.0) 377 : cluster 2021-11-19T08:39:03.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-11-19T08:39:03.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: audit 2021-11-19T08:39:03.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 2021-11-19T08:39: 2021-11-19T08:39:03.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 02.168536+0000 mon.smithi023 (mon. 2021-11-19T08:39:03.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 0) 378 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-19T08:39:03.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 2021-11-19T08:39:03.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: audit 2021-11-19T08:39:03.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 2021-11-19T08: 2021-11-19T08:39:03.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 39:02.169283+0000 mon.smithi023 2021-11-19T08:39:03.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: (mon.0) 379 : audit 2021-11-19T08:39:03.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2021-11-19T08:39:03.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: audit 2021 2021-11-19T08:39:03.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: -11-19T08:39:02 2021-11-19T08:39:03.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: .169373+0000 mon.smithi134 (mon. 2021-11-19T08:39:03.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 1) 9 : audit [INF] 2021-11-19T08:39:03.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: from='osd.4 [v2:172.21.15.134:6808/1867765886,v1:172.21.15.134:6809/1867765886]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2021-11-19T08:39:03.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: audit 2021-11-19T08:39:02.730565+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:39:03.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 133 : audit [DBG] from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:39:03.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: audit 2021 2021-11-19T08:39:03.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: -11-19T08:39:02 2021-11-19T08:39:03.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: .731385+0000 mon.smithi023 2021-11-19T08:39:03.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: (mon.0) 380 : audit 2021-11-19T08:39:03.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-11-19T08:39:03.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: cephadm 2021-11-19T08:39:03.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 2021-11-19T08:39 2021-11-19T08:39:03.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: :02.732562+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:39:03.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: mgr.14164) 134 : cephadm 2021-11-19T08:39:03.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: [INF] Applying service osd.None on host smithi134... 2021-11-19T08:39:03.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: audit 2021-11-19T08:39:03.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 2021-11-19T08:39:03.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 2021-11-19T08 2021-11-19T08:39:03.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: :39:02. 2021-11-19T08:39:03.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 732791+0000 mon.smithi023 (mon. 2021-11-19T08:39:03.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 0) 381 : 2021-11-19T08:39:03.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-11-19T08:39:03.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: audit 2021 2021-11-19T08:39:03.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: -11- 2021-11-19T08:39:03.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 19T08:39:02. 2021-11-19T08:39:03.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 733381+0000 mon.smithi023 (mon. 2021-11-19T08:39:03.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 0) 382 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:39:03.652 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 2021-11-19T08:39:03.652 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: audit 2021-11-19T08:39:03.652 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: 2021-11-19T08:39 2021-11-19T08:39:03.652 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: :02.923367+0000 mon.smithi023 ( 2021-11-19T08:39:03.653 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: mon.0) 383 : audit 2021-11-19T08:39:03.653 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:03 smithi023 conmon[33490]: [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:04.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:04 smithi134 conmon[36363]: cluster 2021-11-19T08:39:02. 2021-11-19T08:39:04.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:04 smithi134 conmon[36363]: 810935+0000 mgr.smithi023.ehlpcl (mgr.14164) 135 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-19T08:39:04.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:04 smithi134 conmon[36363]: audit 2021-11-19T08:39:03.173961+0000 mon.smithi023 (mon.0) 384 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2021-11-19T08:39:04.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:04 smithi134 conmon[36363]: cluster 2021-11-19T08:39:03.174028+0000 mon.smithi023 (mon.0) 385 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-11-19T08:39:04.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:04 smithi134 conmon[36363]: audit 2021-11-19T08:39:03.174439+0000 mon.smithi023 (mon.0) 386 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-19T08:39:04.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:04 smithi134 conmon[36363]: audit 2021-11-19T08:39:03.179464+0000 mon.smithi023 (mon.0) 387 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-19T08:39:04.292 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:04 smithi134 conmon[36363]: cluster 2021-11-19T08:39:04.170847+0000 mon.smithi023 (mon.0) 388 : cluster [INF] osd.4 [v2:172.21.15.134:6808/1867765886,v1:172.21.15.134:6809/1867765886] boot 2021-11-19T08:39:04.293 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:04 smithi134 conmon[36363]: cluster 2021-11-19T08:39:04.170881+0000 mon.smithi023 (mon.0) 389 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-11-19T08:39:04.293 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:04 smithi134 conmon[36363]: audit 2021-11-19T08:39:04.171028+0000 mon.smithi023 (mon.0) 390 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-19T08:39:04.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: cluster 2021-11-19T08:39:02.810935+0000 mgr.smithi023.ehlpcl (mgr.14164) 135 : cluster 2021-11-19T08:39:04.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-11-19T08:39:04.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: audit 2021-11-19T08:39:03.173961+0000 mon.smithi023 (mon.0) 384 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2021-11-19T08:39:04.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: cluster 2021-11-19T08:39:03.174028+0000 mon.smithi023 (mon.0) 385 : cluster [DBG] 2021-11-19T08:39:04.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: osdmap e30: 5 total, 4 up, 5 in 2021-11-19T08:39:04.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:39:04.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: T08:39:03.174439+0000 mon.smithi023 (mon.0) 2021-11-19T08:39:04.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: 386 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-19T08:39:04.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: audit 2021-11-19T08 2021-11-19T08:39:04.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: :39:03.179464+0000 mon.smithi023 (mon.0) 387 : audit [DBG] 2021-11-19T08:39:04.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-19T08:39:04.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: cluster 2021-11-19T08:39:04.170847+0000 2021-11-19T08:39:04.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: mon.smithi023 (mon.0) 388 : cluster [INF] osd.4 [v2:172.21.15.134:6808/1867765886,v1:172.21.15.134:6809/1867765886] boot 2021-11-19T08:39:04.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: 2021-11-19T08:39:04.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: cluster 2021-11-19T08:39:04 2021-11-19T08:39:04.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: .170881+0000 mon.smithi023 (mon.0) 389 : cluster 2021-11-19T08:39:04.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-11-19T08:39:04.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: audit 2021-11-19T 2021-11-19T08:39:04.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: 08:39:04.171028+0000 mon.smithi023 (mon.0 2021-11-19T08:39:04.397 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:04 smithi023 conmon[33490]: ) 390 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-11-19T08:39:05.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: cluster 2021-11-19T08 2021-11-19T08:39:05.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: :39:03.121789+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-19T08:39:05.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: cluster 2021-11-19T08 2021-11-19T08:39:05.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: :39:03.121897+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-19T08:39:05.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: cephadm 2021- 2021-11-19T08:39:05.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: 11-19T08:39:04.617843+0000 2021-11-19T08:39:05.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 136 : cephadm [INF] 2021-11-19T08:39:05.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: Detected new or changed devices on smithi134 2021-11-19T08:39:05.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: audit 2021-11-19T08:39:04 2021-11-19T08:39:05.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: .623215+0000 mon.smithi023 (mon.0) 391 : 2021-11-19T08:39:05.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:05.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: audit 2021-11-19T08 2021-11-19T08:39:05.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: :39:04.632103+0000 mon.smithi023 ( 2021-11-19T08:39:05.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: mon.0) 392 : audit [INF] 2021-11-19T08:39:05.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:05.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: audit 2021-11-19T08:39 2021-11-19T08:39:05.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: :04.635893+0000 mon.smithi023 (mon.0) 2021-11-19T08:39:05.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: 393 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:39:05.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: audit 2021-11-19T 2021-11-19T08:39:05.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: 08:39:05.162385+0000 mon.smithi023 (mon 2021-11-19T08:39:05.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: .0) 394 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86ec41d3-38df-4ee0-a320-402055f7895b"}]: dispatch 2021-11-19T08:39:05.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: audit 2021-11-19T08:39:05.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: 2021-11-19T08:39: 2021-11-19T08:39:05.898 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: 05.162402+0000 mon.smithi134 (mon.1) 2021-11-19T08:39:05.898 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: 10 : audit [INF] from='client.? 172.21.15.134:0/1391837607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86ec41d3-38df-4ee0-a320-402055f7895b"}]: dispatch 2021-11-19T08:39:05.898 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:39:05.898 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: T08:39:05.170170+0000 mon.smithi023 2021-11-19T08:39:05.899 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: (mon.0) 395 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86ec41d3-38df-4ee0-a320-402055f7895b"}]': finished 2021-11-19T08:39:05.899 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: 2021-11-19T08:39:05.899 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: cluster 2021 2021-11-19T08:39:05.899 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: -11-19T08:39:05.170226+0000 2021-11-19T08:39:05.900 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: mon.smithi023 (mon.0) 396 : cluster [DBG] osdmap e32: 6 total, 5 up, 6 in 2021-11-19T08:39:05.900 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: audit 2021-11- 2021-11-19T08:39:05.900 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: 19T08:39:05.170330+0000 mon.smithi023 (mon.0) 397 : audit 2021-11-19T08:39:05.900 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-19T08:39:05.900 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: audit 2021-11-19T08:39:05.901 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: 2021-11-19T08:39:05.568457+0000 mon.smithi134 (mon.1) 11 : audit 2021-11-19T08:39:05.901 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:05 smithi023 conmon[33490]: [DBG] from='client.? 172.21.15.134:0/956972582' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-19T08:39:06.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:05 smithi134 conmon[36363]: cluster 2021-11-19T08:39:03.121789+0000 osd.4 (osd.4) 1 2021-11-19T08:39:06.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:05 smithi134 conmon[36363]: : cluster [DBG] purged_snaps scrub starts 2021-11-19T08:39:06.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:05 smithi134 conmon[36363]: cluster 2021-11-19T08:39:03.121897+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-19T08:39:06.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:05 smithi134 conmon[36363]: cephadm 2021-11-19T08:39:04.617843+0000 mgr.smithi023.ehlpcl (mgr.14164) 136 : cephadm [INF] Detected new or changed devices on smithi134 2021-11-19T08:39:06.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:05 smithi134 conmon[36363]: audit 2021-11-19T08:39:04.623215+0000 mon.smithi023 (mon.0) 391 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:06.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:05 smithi134 conmon[36363]: audit 2021-11-19T08:39:04.632103+0000 mon.smithi023 (mon.0) 392 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:06.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:05 smithi134 conmon[36363]: audit 2021-11-19T08:39:04.635893+0000 mon.smithi023 (mon.0) 393 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:39:06.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:05 smithi134 conmon[36363]: audit 2021-11-19T08:39:05.162385+0000 mon.smithi023 (mon.0) 394 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86ec41d3-38df-4ee0-a320-402055f7895b"}]: dispatch 2021-11-19T08:39:06.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:05 smithi134 conmon[36363]: audit 2021-11-19T08:39:05.162402+0000 mon.smithi134 (mon.1) 10 : audit [INF] from='client.? 172.21.15.134:0/1391837607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86ec41d3-38df-4ee0-a320-402055f7895b"}]: dispatch 2021-11-19T08:39:06.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:05 smithi134 conmon[36363]: audit 2021-11-19T08:39:05.170170+0000 mon.smithi023 (mon.0) 395 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86ec41d3-38df-4ee0-a320-402055f7895b"}]': finished 2021-11-19T08:39:06.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:05 smithi134 conmon[36363]: cluster 2021-11-19T08:39:05.170226+0000 mon.smithi023 (mon.0) 396 : cluster [DBG] osdmap e32: 6 total, 5 up, 6 in 2021-11-19T08:39:06.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:05 smithi134 conmon[36363]: audit 2021-11-19T08:39:05.170330+0000 mon.smithi023 (mon.0) 397 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-19T08:39:06.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:05 smithi134 conmon[36363]: audit 2021-11-19T08:39:05.568457+0000 mon.smithi134 (mon.1) 11 : audit [DBG] from='client.? 172.21.15.134:0/956972582' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-11-19T08:39:06.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:06 smithi023 conmon[33490]: cluster 2021-11-19T08:39:04.811238+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:39:06.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:06 smithi023 conmon[33490]: 14164) 137 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:07.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:06 smithi134 conmon[36363]: cluster 2021-11-19T08:39:04.811238+0000 mgr.smithi023.ehlpcl (mgr.14164) 137 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:08.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:08 smithi023 conmon[33490]: cluster 2021-11-19T08:39:06.811525+0000 mgr.smithi023.ehlpcl (mgr.14164) 138 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:08.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:08 smithi023 conmon[33490]: 2021-11-19T08:39:08.938 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:08 smithi134 conmon[36363]: cluster 2021-11-19T08:39:06.811525+0000 mgr.smithi023.ehlpcl (mgr.14164) 138 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:10.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:10 smithi134 conmon[36363]: cluster 2021-11-19T08:39:08.811916+0000 mgr.smithi023.ehlpcl (mgr.14164) 139 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:10.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:10 smithi134 conmon[36363]: audit 2021-11-19T08:39:09.472160+0000 mon.smithi023 (mon.0) 398 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:10.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:10 smithi134 conmon[36363]: audit 2021-11-19T08:39:09.479665+0000 mon.smithi023 (mon.0) 399 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:10.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:10 smithi023 conmon[33490]: cluster 2021-11-19T08:39:08.811916+0000 mgr.smithi023.ehlpcl 2021-11-19T08:39:10.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:10 smithi023 conmon[33490]: (mgr.14164) 139 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:10.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:10 smithi023 conmon[33490]: audit 2021-11-19T08:39:09.472160+0000 mon.smithi023 (mon.0) 398 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:10.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:10 smithi023 conmon[33490]: audit 2021-11-19T08:39:09.479665+0000 mon.smithi023 (mon.0) 399 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:11.780 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:11 smithi134 conmon[36363]: audit 2021-11-19T08:39:11.230219+0000 mon.smithi023 (mon.0) 400 : audit [INF] 2021-11-19T08:39:11.781 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:11 smithi134 conmon[36363]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-11-19T08:39:11.781 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:11 smithi134 conmon[36363]: audit 2021-11-19T08:39:11.231130+0000 mon.smithi023 (mon.0) 401 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:39:11.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:11 smithi023 conmon[33490]: audit 2021-11-19T08:39:11.230219+0000 mon.smithi023 (mon.0) 400 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-11-19T08:39:11.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:11 smithi023 conmon[33490]: audit 2021-11-19T 2021-11-19T08:39:11.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:11 smithi023 conmon[33490]: 08:39:11.231130+0000 mon.smithi023 (mon.0) 401 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:39:12.650 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:12 smithi134 conmon[36363]: cluster 2021-11-19T08:39:10.812350+0000 mgr.smithi023.ehlpcl (mgr.14164) 140 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:12.650 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:12 smithi134 conmon[36363]: cephadm 2021-11-19T08:39:11.231866+0000 mgr.smithi023.ehlpcl (mgr.14164) 141 : cephadm [INF] Deploying daemon osd.5 on smithi134 2021-11-19T08:39:12.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:12 smithi023 conmon[33490]: cluster 2021-11-19T08:39:10.812350+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:39:12.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:12 smithi023 conmon[33490]: 140 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:12.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:12 smithi023 conmon[33490]: cephadm 2021-11-19T08:39:11.231866+0000 mgr.smithi023.ehlpcl (mgr.14164) 141 : cephadm [INF] Deploying daemon osd.5 on smithi134 2021-11-19T08:39:14.431 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 5 on host 'smithi134' 2021-11-19T08:39:14.432 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:14.444+0000 7f37777fe700 1 -- 172.21.15.134:0/2470521393 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f378000f030 con 0x7f377805c2b0 2021-11-19T08:39:14.432 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:14.446+0000 7f3795083700 1 -- 172.21.15.134:0/2470521393 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f377805c2b0 msgr2=0x7f377805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:14.433 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:14.446+0000 7f3795083700 1 --2- 172.21.15.134:0/2470521393 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f377805c2b0 0x7f377805e770 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f3780005010 tx=0x7f3780016040).stop 2021-11-19T08:39:14.433 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:14.446+0000 7f3795083700 1 -- 172.21.15.134:0/2470521393 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37900f6ff0 msgr2=0x7f379010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:14.433 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:14.446+0000 7f3795083700 1 --2- 172.21.15.134:0/2470521393 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37900f6ff0 0x7f379010da40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f378400e7f0 tx=0x7f378400b040).stop 2021-11-19T08:39:14.434 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:14.446+0000 7f3795083700 1 -- 172.21.15.134:0/2470521393 shutdown_connections 2021-11-19T08:39:14.434 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:14.446+0000 7f3795083700 1 --2- 172.21.15.134:0/2470521393 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f377805c2b0 0x7f377805e770 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:14.434 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:14.446+0000 7f3795083700 1 --2- 172.21.15.134:0/2470521393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37900f66e0 0x7f379010d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:14.434 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:14.446+0000 7f3795083700 1 --2- 172.21.15.134:0/2470521393 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37900f6ff0 0x7f379010da40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:14.435 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:14.446+0000 7f3795083700 1 -- 172.21.15.134:0/2470521393 >> 172.21.15.134:0/2470521393 conn(0x7f37900f1f00 msgr2=0x7f37900fb470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:14.435 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:14.447+0000 7f3795083700 1 -- 172.21.15.134:0/2470521393 shutdown_connections 2021-11-19T08:39:14.435 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:14.447+0000 7f3795083700 1 -- 172.21.15.134:0/2470521393 wait complete. 2021-11-19T08:39:14.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:14 smithi134 conmon[36363]: cluster 2021-11-19T08:39:12. 2021-11-19T08:39:14.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:14 smithi134 conmon[36363]: 812764+0000 mgr.smithi023.ehlpcl (mgr.14164) 142 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:14.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:14 smithi134 conmon[36363]: audit 2021-11-19T08:39:14.431520+0000 mon.smithi023 (mon.0) 402 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:39:14.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:14 smithi134 conmon[36363]: audit 2021-11-19T 2021-11-19T08:39:14.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:14 smithi134 conmon[36363]: 08:39:14.436940+0000 mon.smithi023 (mon.0) 403 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:14.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:14 smithi134 conmon[36363]: audit 2021-11-19T08 2021-11-19T08:39:14.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:14 smithi134 conmon[36363]: :39:14.444341+0000 mon.smithi023 (mon.0 2021-11-19T08:39:14.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:14 smithi134 conmon[36363]: ) 404 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:14.862 DEBUG:teuthology.orchestra.run.smithi134:osd.5> sudo journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.5.service 2021-11-19T08:39:14.864 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2021-11-19T08:39:14.865 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph osd stat -f json 2021-11-19T08:39:14.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:14 smithi023 conmon[33490]: cluster 2021-11-19T08:39:12.812764+0000 mgr.smithi023.ehlpcl (mgr.14164) 142 2021-11-19T08:39:14.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:14 smithi023 conmon[33490]: : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:14.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:14 smithi023 conmon[33490]: audit 2021-11-19T08:39:14.431520+0000 mon.smithi023 (mon.0) 402 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:39:14.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:14 smithi023 conmon[33490]: audit 2021-11-19T08:39:14.436940+0000 mon.smithi023 (mon. 2021-11-19T08:39:14.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:14 smithi023 conmon[33490]: 0) 403 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:14.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:14 smithi023 conmon[33490]: audit 2021-11-19T08 2021-11-19T08:39:14.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:14 smithi023 conmon[33490]: :39:14.444341+0000 mon.smithi023 (mon. 2021-11-19T08:39:14.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:14 smithi023 conmon[33490]: 0) 404 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:14.894 INFO:journalctl@ceph.osd.5.smithi134.stdout:-- Logs begin at Fri 2021-11-19 08:24:18 UTC. -- 2021-11-19T08:39:16.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:16 smithi134 conmon[36363]: cluster 2021-11-19T08:39:14.813286+0000 2021-11-19T08:39:16.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:16 smithi134 conmon[36363]: mgr.smithi023.ehlpcl (mgr.14164) 143 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:16.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:16 smithi023 conmon[33490]: cluster 2021-11-19T08:39:14.813286+0000 2021-11-19T08:39:16.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:16 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 143 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:16.900 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.909+0000 7f93df59e700 1 -- 172.21.15.23:0/2369918121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93e00f66e0 msgr2=0x7f93e00f6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:16.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.909+0000 7f93df59e700 1 --2- 172.21.15.23:0/2369918121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93e00f66e0 0x7f93e00f6b40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f93c8004660 tx=0x7f93c8009b30).stop 2021-11-19T08:39:16.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.909+0000 7f93df59e700 1 -- 172.21.15.23:0/2369918121 shutdown_connections 2021-11-19T08:39:16.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.909+0000 7f93df59e700 1 --2- 172.21.15.23:0/2369918121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93e00f66e0 0x7f93e00f6b40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:16.902 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.909+0000 7f93df59e700 1 --2- 172.21.15.23:0/2369918121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93e00f5440 0x7f93e00f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:16.902 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.909+0000 7f93df59e700 1 -- 172.21.15.23:0/2369918121 >> 172.21.15.23:0/2369918121 conn(0x7f93e00f0ba0 msgr2=0x7f93e00f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:16.902 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.909+0000 7f93df59e700 1 -- 172.21.15.23:0/2369918121 shutdown_connections 2021-11-19T08:39:16.902 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.909+0000 7f93df59e700 1 -- 172.21.15.23:0/2369918121 wait complete. 2021-11-19T08:39:16.903 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.910+0000 7f93df59e700 1 Processor -- start 2021-11-19T08:39:16.903 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.910+0000 7f93df59e700 1 -- start start 2021-11-19T08:39:16.903 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.910+0000 7f93df59e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93e00f5440 0x7f93e0108f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:16.904 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.910+0000 7f93df59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93e00f66e0 0x7f93e0109460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:16.904 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.910+0000 7f93df59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e0109a10 con 0x7f93e00f66e0 2021-11-19T08:39:16.904 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.910+0000 7f93df59e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e0109b50 con 0x7f93e00f5440 2021-11-19T08:39:16.904 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.910+0000 7f93ddd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93e00f66e0 0x7f93e0109460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:16.904 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.910+0000 7f93de59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93e00f5440 0x7f93e0108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.910+0000 7f93ddd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93e00f66e0 0x7f93e0109460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46606/0 (socket says 172.21.15.23:46606) 2021-11-19T08:39:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.910+0000 7f93ddd9b700 1 -- 172.21.15.23:0/3947073927 learned_addr learned my addr 172.21.15.23:0/3947073927 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.911+0000 7f93ddd9b700 1 -- 172.21.15.23:0/3947073927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93e00f5440 msgr2=0x7f93e0108f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.911+0000 7f93ddd9b700 1 --2- 172.21.15.23:0/3947073927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93e00f5440 0x7f93e0108f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.911+0000 7f93ddd9b700 1 -- 172.21.15.23:0/3947073927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c8005040 con 0x7f93e00f66e0 2021-11-19T08:39:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.911+0000 7f93ddd9b700 1 --2- 172.21.15.23:0/3947073927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93e00f66e0 0x7f93e0109460 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f93c8009f50 tx=0x7f93c8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.911+0000 7f93d6ffd700 1 -- 172.21.15.23:0/3947073927 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c8018070 con 0x7f93e00f66e0 2021-11-19T08:39:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.911+0000 7f93df59e700 1 -- 172.21.15.23:0/3947073927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e010a490 con 0x7f93e00f66e0 2021-11-19T08:39:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.911+0000 7f93d6ffd700 1 -- 172.21.15.23:0/3947073927 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93c8013410 con 0x7f93e00f66e0 2021-11-19T08:39:16.907 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.911+0000 7f93d6ffd700 1 -- 172.21.15.23:0/3947073927 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c801c3c0 con 0x7f93e00f66e0 2021-11-19T08:39:16.907 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.911+0000 7f93df59e700 1 -- 172.21.15.23:0/3947073927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e010a9f0 con 0x7f93e00f66e0 2021-11-19T08:39:16.907 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.912+0000 7f93df59e700 1 -- 172.21.15.23:0/3947073927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e0045ad0 con 0x7f93e00f66e0 2021-11-19T08:39:16.908 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.912+0000 7f93d6ffd700 1 -- 172.21.15.23:0/3947073927 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f93c801d420 con 0x7f93e00f66e0 2021-11-19T08:39:16.908 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.913+0000 7f93d6ffd700 1 --2- 172.21.15.23:0/3947073927 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f93cc05c300 0x7f93cc05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:16.908 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.913+0000 7f93d6ffd700 1 -- 172.21.15.23:0/3947073927 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4235+0+0 (secure 0 0 0) 0x7f93c80776c0 con 0x7f93e00f66e0 2021-11-19T08:39:16.908 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.913+0000 7f93de59c700 1 --2- 172.21.15.23:0/3947073927 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f93cc05c300 0x7f93cc05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:16.908 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.913+0000 7f93de59c700 1 --2- 172.21.15.23:0/3947073927 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f93cc05c300 0x7f93cc05e7c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f93e00f6540 tx=0x7f93d000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:16.909 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:16.917+0000 7f93d6ffd700 1 -- 172.21.15.23:0/3947073927 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93c804a800 con 0x7f93e00f66e0 2021-11-19T08:39:17.061 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:17.076+0000 7f93df59e700 1 -- 172.21.15.23:0/3947073927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f93e010f590 con 0x7f93e00f66e0 2021-11-19T08:39:17.062 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:17.077+0000 7f93d6ffd700 1 -- 172.21.15.23:0/3947073927 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v32) v1 ==== 74+0+130 (secure 0 0 0) 0x7f93c804de20 con 0x7f93e00f66e0 2021-11-19T08:39:17.063 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:39:17.065 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:17.078+0000 7f93df59e700 1 -- 172.21.15.23:0/3947073927 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f93cc05c300 msgr2=0x7f93cc05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:17.065 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:17.078+0000 7f93df59e700 1 --2- 172.21.15.23:0/3947073927 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f93cc05c300 0x7f93cc05e7c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f93e00f6540 tx=0x7f93d000b040).stop 2021-11-19T08:39:17.065 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:17.078+0000 7f93df59e700 1 -- 172.21.15.23:0/3947073927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93e00f66e0 msgr2=0x7f93e0109460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:17.066 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:17.078+0000 7f93df59e700 1 --2- 172.21.15.23:0/3947073927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93e00f66e0 0x7f93e0109460 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f93c8009f50 tx=0x7f93c8009b30).stop 2021-11-19T08:39:17.066 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:17.079+0000 7f93df59e700 1 -- 172.21.15.23:0/3947073927 shutdown_connections 2021-11-19T08:39:17.066 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:17.079+0000 7f93df59e700 1 --2- 172.21.15.23:0/3947073927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93e00f5440 0x7f93e0108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:17.066 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:17.079+0000 7f93df59e700 1 --2- 172.21.15.23:0/3947073927 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f93cc05c300 0x7f93cc05e7c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:17.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:17.079+0000 7f93df59e700 1 --2- 172.21.15.23:0/3947073927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93e00f66e0 0x7f93e0109460 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:17.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:17.079+0000 7f93df59e700 1 -- 172.21.15.23:0/3947073927 >> 172.21.15.23:0/3947073927 conn(0x7f93e00f0ba0 msgr2=0x7f93e005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:17.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:17.079+0000 7f93df59e700 1 -- 172.21.15.23:0/3947073927 shutdown_connections 2021-11-19T08:39:17.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:17.079+0000 7f93df59e700 1 -- 172.21.15.23:0/3947073927 wait complete. 2021-11-19T08:39:17.758 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:17 smithi134 conmon[36363]: audit 2021-11-19T08:39:17.078034+0000 mon.smithi023 (mon.0 2021-11-19T08:39:17.759 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:17 smithi134 conmon[36363]: ) 405 : audit [DBG] from='client.? 172.21.15.23:0/3947073927' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2021-11-19T08:39:17.759 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:17 smithi134 conmon[36363]: audit 2021-11-19T08:39:17.251429+0000 mon.smithi023 (mon.0) 406 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:17.822 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:17 smithi023 conmon[33490]: audit 2021-11-19T08:39:17.078034 2021-11-19T08:39:17.823 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:17 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 405 : audit [DBG] from='client.? 172.21.15.23:0/3947073927' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2021-11-19T08:39:17.823 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:17 smithi023 conmon[33490]: audit 2021-11-19T08:39:17.251429+0000 mon.smithi023 (mon.0) 406 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:17.845 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":32,"num_osds":6,"num_up_osds":5,"osd_up_since":1637311144,"num_in_osds":6,"osd_in_since":1637311145,"num_remapped_pgs":0} 2021-11-19T08:39:18.490 INFO:journalctl@ceph.osd.5.smithi134.stdout:Nov 19 08:39:18 smithi134 conmon[52519]: debug 2021-11-19T08:39:18.337+0000 7f4a3cce1080 -1 osd.5 0 log_to_monitors {default=true} 2021-11-19T08:39:18.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:18 smithi134 conmon[36363]: cluster 2021-11-19T08:39:16.813641+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:39:18.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:18 smithi134 conmon[36363]: 14164) 144 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:18.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:18 smithi134 conmon[36363]: audit 2021-11-19T08:39:18.341317+0000 mon.smithi023 (mon.0) 407 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-19T08:39:18.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:18 smithi134 conmon[36363]: audit 2021-11-19T08:39:18.341409+0000 mon.smithi134 (mon.1) 12 : audit [INF] from='osd.5 [v2:172.21.15.134:6816/3784199400,v1:172.21.15.134:6817/3784199400]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-19T08:39:18.846 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph osd stat -f json 2021-11-19T08:39:18.874 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:18 smithi023 conmon[33490]: cluster 2021-11-19T08:39:16.813641+0000 mgr.smithi023.ehlpcl 2021-11-19T08:39:18.874 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:18 smithi023 conmon[33490]: (mgr.14164) 144 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:18.874 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:18 smithi023 conmon[33490]: audit 2021-11-19T08:39:18.341317+0000 mon.smithi023 (mon.0) 407 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-19T08:39:18.875 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:18 smithi023 conmon[33490]: audit 2021-11-19T08:39:18.341409+0000 mon.smithi134 (mon.1) 12 : audit [INF] from='osd.5 [v2:172.21.15.134:6816/3784199400,v1:172.21.15.134:6817/3784199400]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-11-19T08:39:19.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:19 smithi023 conmon[33490]: audit 2021-11-19T08:39:18.502306+0000 mon.smithi023 (mon. 2021-11-19T08:39:19.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:19 smithi023 conmon[33490]: 0) 408 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-11-19T08:39:19.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:19 smithi023 conmon[33490]: cluster 2021-11-19T08:39:18.502452+0000 mon.smithi023 (mon.0) 409 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2021-11-19T08:39:19.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:19 smithi023 conmon[33490]: audit 2021-11-19T08:39:18.503072+0000 mon.smithi023 (mon.0) 410 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-19T08:39:19.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:19 smithi023 conmon[33490]: audit 2021-11-19T08:39:18.503370+0000 mon.smithi023 (mon.0) 411 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2021-11-19T08:39:19.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:19 smithi023 conmon[33490]: audit 2021-11-19T08:39:18.503418+0000 mon.smithi134 (mon.1) 13 : audit [INF] from='osd.5 [v2:172.21.15.134:6816/3784199400,v1:172.21.15.134:6817/3784199400]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2021-11-19T08:39:19.770 INFO:journalctl@ceph.osd.5.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[52519]: debug 2021-11-19T08:39:19.511+0000 7f4a33252700 -1 osd.5 0 waiting for initial osdmap 2021-11-19T08:39:19.771 INFO:journalctl@ceph.osd.5.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[52519]: debug 2021-11-19T08:39:19.520+0000 7f4a2c9c8700 -1 osd.5 34 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2021-11-19T08:39:19.771 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: audit 2021-11-19T08:39:18.502306 2021-11-19T08:39:19.771 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 408 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-11-19T08:39:19.772 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: cluster 2021-11-19T08 2021-11-19T08:39:19.772 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: :39:18.502452+0000 mon.smithi023 (mon.0) 409 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2021-11-19T08:39:19.772 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: audit 2021-11-19T08:39:19.772 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: 2021-11-19T08:39: 2021-11-19T08:39:19.772 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: 18.503072+0000 mon.smithi023 (mon.0) 410 : 2021-11-19T08:39:19.773 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-19T08:39:19.773 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: audit 2021-11- 2021-11-19T08:39:19.774 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: 19T08:39:18.503370+0000 2021-11-19T08:39:19.774 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: mon.smithi023 (mon.0) 411 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2021-11-19T08:39:19.774 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: 2021-11-19T08:39:19.774 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: audit 2021-11-19T08:39:19.775 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: 2021-11-19T08:39:18. 2021-11-19T08:39:19.775 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: 503418+0000 mon.smithi134 (mon.1) 13 : audit 2021-11-19T08:39:19.775 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:19 smithi134 conmon[36363]: [INF] from='osd.5 [v2:172.21.15.134:6816/3784199400,v1:172.21.15.134:6817/3784199400]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2021-11-19T08:39:20.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.470+0000 7fadd249c700 1 -- 172.21.15.23:0/3561949062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadcc0f66e0 msgr2=0x7fadcc0f6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:20.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.470+0000 7fadd249c700 1 --2- 172.21.15.23:0/3561949062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadcc0f66e0 0x7fadcc0f6b40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fadc0004660 tx=0x7fadc0009b30).stop 2021-11-19T08:39:20.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.470+0000 7fadd249c700 1 -- 172.21.15.23:0/3561949062 shutdown_connections 2021-11-19T08:39:20.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.470+0000 7fadd249c700 1 --2- 172.21.15.23:0/3561949062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadcc0f66e0 0x7fadcc0f6b40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:20.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.470+0000 7fadd249c700 1 --2- 172.21.15.23:0/3561949062 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fadcc0f5440 0x7fadcc0f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:20.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.470+0000 7fadd249c700 1 -- 172.21.15.23:0/3561949062 >> 172.21.15.23:0/3561949062 conn(0x7fadcc0f0ba0 msgr2=0x7fadcc0f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:20.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.470+0000 7fadd249c700 1 -- 172.21.15.23:0/3561949062 shutdown_connections 2021-11-19T08:39:20.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.470+0000 7fadd249c700 1 -- 172.21.15.23:0/3561949062 wait complete. 2021-11-19T08:39:20.460 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.471+0000 7fadd249c700 1 Processor -- start 2021-11-19T08:39:20.460 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.471+0000 7fadd249c700 1 -- start start 2021-11-19T08:39:20.460 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.471+0000 7fadd249c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadcc0f5440 0x7fadcc109990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:20.460 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.471+0000 7fadd249c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fadcc0f66e0 0x7fadcc10be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:20.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.471+0000 7fadd249c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc060420 con 0x7fadcc0f5440 2021-11-19T08:39:20.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.471+0000 7fadd249c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadcc060560 con 0x7fadcc0f66e0 2021-11-19T08:39:20.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.471+0000 7fadd149a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadcc0f5440 0x7fadcc109990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:20.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.471+0000 7fadd149a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadcc0f5440 0x7fadcc109990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46614/0 (socket says 172.21.15.23:46614) 2021-11-19T08:39:20.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.471+0000 7fadd149a700 1 -- 172.21.15.23:0/2889587708 learned_addr learned my addr 172.21.15.23:0/2889587708 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:20.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.471+0000 7fadd0c99700 1 --2- 172.21.15.23:0/2889587708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fadcc0f66e0 0x7fadcc10be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:20.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.472+0000 7fadd149a700 1 -- 172.21.15.23:0/2889587708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fadcc0f66e0 msgr2=0x7fadcc10be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:20.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.472+0000 7fadd149a700 1 --2- 172.21.15.23:0/2889587708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fadcc0f66e0 0x7fadcc10be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:20.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.472+0000 7fadd149a700 1 -- 172.21.15.23:0/2889587708 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc0005040 con 0x7fadcc0f5440 2021-11-19T08:39:20.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.472+0000 7fadd149a700 1 --2- 172.21.15.23:0/2889587708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadcc0f5440 0x7fadcc109990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fadc400e7c0 tx=0x7fadc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:20.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.472+0000 7fadc9ffb700 1 -- 172.21.15.23:0/2889587708 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadc40092e0 con 0x7fadcc0f5440 2021-11-19T08:39:20.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.472+0000 7fadc9ffb700 1 -- 172.21.15.23:0/2889587708 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fadc400f040 con 0x7fadcc0f5440 2021-11-19T08:39:20.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.472+0000 7fadc9ffb700 1 -- 172.21.15.23:0/2889587708 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadc40147c0 con 0x7fadcc0f5440 2021-11-19T08:39:20.464 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.472+0000 7fadd249c700 1 -- 172.21.15.23:0/2889587708 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadcc10c4a0 con 0x7fadcc0f5440 2021-11-19T08:39:20.464 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.472+0000 7fadd249c700 1 -- 172.21.15.23:0/2889587708 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadcc10c9e0 con 0x7fadcc0f5440 2021-11-19T08:39:20.465 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.474+0000 7fadc9ffb700 1 -- 172.21.15.23:0/2889587708 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fadc401b030 con 0x7fadcc0f5440 2021-11-19T08:39:20.466 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.474+0000 7fadd249c700 1 -- 172.21.15.23:0/2889587708 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadcc045ad0 con 0x7fadcc0f5440 2021-11-19T08:39:20.466 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.474+0000 7fadc9ffb700 1 --2- 172.21.15.23:0/2889587708 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fadb405c300 0x7fadb405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:20.466 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.474+0000 7fadc9ffb700 1 -- 172.21.15.23:0/2889587708 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4272+0+0 (secure 0 0 0) 0x7fadc40759f0 con 0x7fadcc0f5440 2021-11-19T08:39:20.466 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.475+0000 7fadd0c99700 1 --2- 172.21.15.23:0/2889587708 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fadb405c300 0x7fadb405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:20.466 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.475+0000 7fadd0c99700 1 --2- 172.21.15.23:0/2889587708 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fadb405c300 0x7fadb405e7c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fadc0005010 tx=0x7fadc0016040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:20.467 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.479+0000 7fadc9ffb700 1 -- 172.21.15.23:0/2889587708 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fadc4077e60 con 0x7fadcc0f5440 2021-11-19T08:39:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.643+0000 7fadd249c700 1 -- 172.21.15.23:0/2889587708 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fadcc027a00 con 0x7fadcc0f5440 2021-11-19T08:39:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.643+0000 7fadc9ffb700 1 -- 172.21.15.23:0/2889587708 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v35) v1 ==== 74+0+130 (secure 0 0 0) 0x7fadc4019070 con 0x7fadcc0f5440 2021-11-19T08:39:20.630 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:39:20.631 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.645+0000 7fadd249c700 1 -- 172.21.15.23:0/2889587708 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fadb405c300 msgr2=0x7fadb405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:20.631 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.645+0000 7fadd249c700 1 --2- 172.21.15.23:0/2889587708 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fadb405c300 0x7fadb405e7c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fadc0005010 tx=0x7fadc0016040).stop 2021-11-19T08:39:20.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.645+0000 7fadd249c700 1 -- 172.21.15.23:0/2889587708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadcc0f5440 msgr2=0x7fadcc109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:20.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.645+0000 7fadd249c700 1 --2- 172.21.15.23:0/2889587708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadcc0f5440 0x7fadcc109990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fadc400e7c0 tx=0x7fadc400b040).stop 2021-11-19T08:39:20.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.645+0000 7fadd249c700 1 -- 172.21.15.23:0/2889587708 shutdown_connections 2021-11-19T08:39:20.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.645+0000 7fadd249c700 1 --2- 172.21.15.23:0/2889587708 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fadb405c300 0x7fadb405e7c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:20.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.645+0000 7fadd249c700 1 --2- 172.21.15.23:0/2889587708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadcc0f5440 0x7fadcc109990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.645+0000 7fadd249c700 1 --2- 172.21.15.23:0/2889587708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fadcc0f66e0 0x7fadcc10be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.645+0000 7fadd249c700 1 -- 172.21.15.23:0/2889587708 >> 172.21.15.23:0/2889587708 conn(0x7fadcc0f0ba0 msgr2=0x7fadcc0fa130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.646+0000 7fadd249c700 1 -- 172.21.15.23:0/2889587708 shutdown_connections 2021-11-19T08:39:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:20.646+0000 7fadd249c700 1 -- 172.21.15.23:0/2889587708 wait complete. 2021-11-19T08:39:20.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:20 smithi134 conmon[36363]: cluster 2021-11-19T08:39:18.813890+0000 mgr.smithi023.ehlpcl (mgr.14164) 145 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:20.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:20 smithi134 conmon[36363]: audit 2021-11-19T08 2021-11-19T08:39:20.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:20 smithi134 conmon[36363]: :39:19.505223+0000 mon.smithi023 (mon.0) 412 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2021-11-19T08:39:20.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:20 smithi134 conmon[36363]: cluster 2021-11-19T08:39:19.505337+0000 mon.smithi023 (mon.0) 413 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2021-11-19T08:39:20.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:20 smithi134 conmon[36363]: audit 2021-11-19T08:39:19.505494+0000 mon.smithi023 (mon.0) 414 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-19T08:39:20.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:20 smithi134 conmon[36363]: audit 2021-11-19T08:39:19.508710+0000 mon.smithi023 (mon.0) 415 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-19T08:39:20.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:20 smithi134 conmon[36363]: audit 2021-11-19T08:39:19.835791+0000 mon.smithi023 (mon.0) 416 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:20.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:20 smithi134 conmon[36363]: audit 2021-11-19T08:39:20.161726+0000 mon.smithi023 (mon.0) 417 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:20.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:20 smithi134 conmon[36363]: audit 2021-11-19T08:39:20.506383+0000 mon.smithi023 (mon.0) 418 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:20.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:20 smithi134 conmon[36363]: audit 2021-11-19T08:39:20.508642+0000 mon.smithi023 (mon.0) 419 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-19T08:39:20.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:20 smithi134 conmon[36363]: audit 2021-11-19T08:39:20.509561+0000 mon.smithi023 (mon.0) 420 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:39:20.817 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: cluster 2021-11-19T08:39:18.813890+0000 mgr.smithi023.ehlpcl 2021-11-19T08:39:20.817 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: (mgr.14164) 145 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:20.817 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: audit 2021-11-19T08:39:19.505223+0000 mon.smithi023 (mon.0) 412 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2021-11-19T08:39:20.818 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: cluster 2021-11-19T08:39:19.505337+0000 mon.smithi023 (mon.0) 413 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2021-11-19T08:39:20.818 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: audit 2021-11-19T08:39:19.505494+0000 mon.smithi023 (mon.0) 414 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-19T08:39:20.818 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: audit 2021-11-19T08:39:19.508710+0000 mon.smithi023 (mon.0) 415 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-19T08:39:20.819 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: audit 2021-11-19T08:39:19.835791+0000 mon.smithi023 (mon.0) 416 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:20.819 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: audit 2021-11-19T08:39:20.819 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: 2021-11-19T08:39:20.161726+0000 mon.smithi023 (mon.0) 417 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:20.819 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: audit 2021-11-19T08:39:20.820 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: 2021-11-19T08: 2021-11-19T08:39:20.820 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: 39:20.506383+0000 mon.smithi023 (mon. 2021-11-19T08:39:20.820 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: 0) 418 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:20.820 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: audit 2021-11-19T08:39:20.821 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: 2021-11-19T08:39 2021-11-19T08:39:20.821 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: :20.508642+0000 mon.smithi023 (mon.0 2021-11-19T08:39:20.821 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: ) 419 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-19T08:39:20.821 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: audit 2021-11-19T08:39:20.821 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: 2021-11-19T08: 2021-11-19T08:39:20.822 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: 39:20.509561+0000 mon.smithi023 (mon 2021-11-19T08:39:20.822 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:20 smithi023 conmon[33490]: .0) 420 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:39:21.035 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":35,"num_osds":6,"num_up_osds":6,"osd_up_since":1637311160,"num_in_osds":6,"osd_in_since":1637311145,"num_remapped_pgs":0} 2021-11-19T08:39:21.036 INFO:tasks.cephadm:Setting up client nodes... 2021-11-19T08:39:21.037 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-11-19T08:39:21.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: cluster 2021-11-19T08:39:19.383441+0000 osd.5 ( 2021-11-19T08:39:21.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-19T08:39:21.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: cluster 2021-11-19T08:39:19. 2021-11-19T08:39:21.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: 383580+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-19T08:39:21.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: cephadm 2021-11-19 2021-11-19T08:39:21.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: T08:39:20.155583+0000 mgr.smithi023.ehlpcl (mgr.14164) 146 : cephadm [INF] Detected new or changed devices on smithi134 2021-11-19T08:39:21.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: cluster 2021 2021-11-19T08:39:21.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: -11-19T08:39:20.518108 2021-11-19T08:39:21.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 421 : 2021-11-19T08:39:21.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: cluster [INF] osd.5 [v2:172.21.15.134:6816/3784199400,v1:172.21.15.134:6817/3784199400] boot 2021-11-19T08:39:21.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: cluster 2021-11 2021-11-19T08:39:21.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: -19T08:39:20.518156+0000 2021-11-19T08:39:21.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: mon.smithi023 (mon.0) 422 : cluster 2021-11-19T08:39:21.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: [DBG] osdmap e35: 6 total, 6 up, 6 in 2021-11-19T08:39:21.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:39:21.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: 11-19T08:39:20. 2021-11-19T08:39:21.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: 518575+0000 mon.smithi023 (mon.0) 423 2021-11-19T08:39:21.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-19T08:39:21.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:39:21.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: 11-19T08:39:20.644601 2021-11-19T08:39:21.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 424 : 2021-11-19T08:39:21.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: audit [DBG] from='client.? 172.21.15.23:0/2889587708' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2021-11-19T08:39:21.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:39:21.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: 11-19T08:39:20.913025 2021-11-19T08:39:21.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 425 2021-11-19T08:39:21.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:39:21.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:39:21.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: 11-19T08:39:20.913269 2021-11-19T08:39:21.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 426 : 2021-11-19T08:39:21.798 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:21 smithi134 conmon[36363]: audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:39:21.823 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:21 smithi023 conmon[33490]: cluster 2021-11-19T08:39:19.383441+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-11-19T08:39:21.823 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:21 smithi023 conmon[33490]: cluster 2021-11-19T08:39:19.383580+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-11-19T08:39:21.823 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:21 smithi023 conmon[33490]: cephadm 2021-11-19T08:39:20.155583+0000 mgr.smithi023.ehlpcl (mgr.14164) 146 : cephadm [INF] Detected new or changed devices on smithi134 2021-11-19T08:39:21.823 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:21 smithi023 conmon[33490]: cluster 2021-11-19T08:39:20.518108+0000 mon.smithi023 (mon.0) 421 : cluster [INF] osd.5 [v2:172.21.15.134:6816/3784199400,v1:172.21.15.134:6817/3784199400] boot 2021-11-19T08:39:21.824 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:21 smithi023 conmon[33490]: cluster 2021-11-19T08:39:20.518156+0000 2021-11-19T08:39:21.824 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:21 smithi023 conmon[33490]: mon.smithi023 (mon.0) 422 : cluster [DBG] osdmap e35: 6 total, 6 up, 6 in 2021-11-19T08:39:21.824 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:21 smithi023 conmon[33490]: audit 2021-11-19T08:39:20.518575+0000 mon.smithi023 (mon.0) 423 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-11-19T08:39:21.824 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:21 smithi023 conmon[33490]: audit 2021-11-19T08:39:20.644601+0000 mon.smithi023 (mon.0) 424 : audit [DBG] from='client.? 172.21.15.23:0/2889587708' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2021-11-19T08:39:21.825 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:21 smithi023 conmon[33490]: audit 2021-11-19T08:39:20.913025+0000 mon.smithi023 (mon.0) 425 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:39:21.825 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:21 smithi023 conmon[33490]: audit 2021-11-19T08:39:20.913269+0000 mon.smithi023 (mon.0) 426 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:39:22.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:22 smithi023 conmon[33490]: cluster 2021-11-19T08:39:20.814178+0000 mgr.smithi023.ehlpcl (mgr.14164) 147 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:22.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:22 smithi023 conmon[33490]: cluster 2021-11-19T08:39:21.530993+0000 mon.smithi023 (mon.0) 427 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2021-11-19T08:39:22.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:22 smithi134 conmon[36363]: cluster 2021-11-19T08:39:20.814178+0000 mgr.smithi023.ehlpcl (mgr.14164) 147 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-11-19T08:39:22.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:22 smithi134 conmon[36363]: cluster 2021-11-19T08:39:21.530993 2021-11-19T08:39:22.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:22 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 427 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2021-11-19T08:39:22.845 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.857+0000 7f8badf1d700 1 -- 172.21.15.23:0/3654822070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ba80f4800 msgr2=0x7f8ba80f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:22.845 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.857+0000 7f8badf1d700 1 --2- 172.21.15.23:0/3654822070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ba80f4800 0x7f8ba80f4c20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f8b980099e0 tx=0x7f8b98004d10).stop 2021-11-19T08:39:22.845 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.857+0000 7f8badf1d700 1 -- 172.21.15.23:0/3654822070 shutdown_connections 2021-11-19T08:39:22.845 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.857+0000 7f8badf1d700 1 --2- 172.21.15.23:0/3654822070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ba80f51a0 0x7f8ba80f7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-19T08:39:22.846 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.857+0000 7f8badf1d700 1 --2- 172.21.15.23:0/3654822070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ba80f4800 0x7f8ba80f4c20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:22.846 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.857+0000 7f8badf1d700 1 -- 172.21.15.23:0/3654822070 >> 172.21.15.23:0/3654822070 conn(0x7f8ba80f0320 msgr2=0x7f8ba80f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:22.846 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.857+0000 7f8badf1d700 1 -- 172.21.15.23:0/3654822070 shutdown_connections 2021-11-19T08:39:22.847 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.857+0000 7f8badf1d700 1 -- 172.21.15.23:0/3654822070 wait complete. 2021-11-19T08:39:22.847 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.858+0000 7f8badf1d700 1 Processor -- start 2021-11-19T08:39:22.847 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.858+0000 7f8badf1d700 1 -- start start 2021-11-19T08:39:22.847 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.858+0000 7f8badf1d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ba80f4800 0x7f8ba8106f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:22.847 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.858+0000 7f8badf1d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ba80f51a0 0x7f8ba8107460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:22.848 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.858+0000 7f8badf1d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba8107aa0 con 0x7f8ba80f51a0 2021-11-19T08:39:22.848 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.858+0000 7f8badf1d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba8108350 con 0x7f8ba80f4800 2021-11-19T08:39:22.848 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.858+0000 7f8ba7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ba80f51a0 0x7f8ba8107460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:22.848 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.858+0000 7f8ba7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ba80f51a0 0x7f8ba8107460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46640/0 (socket says 172.21.15.23:46640) 2021-11-19T08:39:22.849 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.858+0000 7f8ba7fff700 1 -- 172.21.15.23:0/3700169541 learned_addr learned my addr 172.21.15.23:0/3700169541 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:22.849 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.858+0000 7f8bacf1b700 1 --2- 172.21.15.23:0/3700169541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ba80f4800 0x7f8ba8106f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:22.849 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.859+0000 7f8ba7fff700 1 -- 172.21.15.23:0/3700169541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ba80f4800 msgr2=0x7f8ba8106f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:22.849 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.859+0000 7f8ba7fff700 1 --2- 172.21.15.23:0/3700169541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ba80f4800 0x7f8ba8106f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:22.849 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.859+0000 7f8ba7fff700 1 -- 172.21.15.23:0/3700169541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b98009640 con 0x7f8ba80f51a0 2021-11-19T08:39:22.850 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.859+0000 7f8ba7fff700 1 --2- 172.21.15.23:0/3700169541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ba80f51a0 0x7f8ba8107460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f8b900093f0 tx=0x7f8b90006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:22.850 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.859+0000 7f8ba57fa700 1 -- 172.21.15.23:0/3700169541 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b90013640 con 0x7f8ba80f51a0 2021-11-19T08:39:22.851 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.859+0000 7f8ba57fa700 1 -- 172.21.15.23:0/3700169541 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b90013c50 con 0x7f8ba80f51a0 2021-11-19T08:39:22.851 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.859+0000 7f8badf1d700 1 -- 172.21.15.23:0/3700169541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba8108630 con 0x7f8ba80f51a0 2021-11-19T08:39:22.851 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.859+0000 7f8ba57fa700 1 -- 172.21.15.23:0/3700169541 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b90012c80 con 0x7f8ba80f51a0 2021-11-19T08:39:22.852 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.859+0000 7f8badf1d700 1 -- 172.21.15.23:0/3700169541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba8108b70 con 0x7f8ba80f51a0 2021-11-19T08:39:22.853 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.860+0000 7f8ba57fa700 1 -- 172.21.15.23:0/3700169541 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8b9001a060 con 0x7f8ba80f51a0 2021-11-19T08:39:22.853 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.860+0000 7f8badf1d700 1 -- 172.21.15.23:0/3700169541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba8059100 con 0x7f8ba80f51a0 2021-11-19T08:39:22.853 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.861+0000 7f8ba57fa700 1 --2- 172.21.15.23:0/3700169541 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8b9405c300 0x7f8b9405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:22.854 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.861+0000 7f8ba57fa700 1 -- 172.21.15.23:0/3700169541 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8b90075ba0 con 0x7f8ba80f51a0 2021-11-19T08:39:22.854 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.861+0000 7f8bacf1b700 1 --2- 172.21.15.23:0/3700169541 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8b9405c300 0x7f8b9405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:22.854 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.861+0000 7f8bacf1b700 1 --2- 172.21.15.23:0/3700169541 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8b9405c300 0x7f8b9405e7c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f8b9800b620 tx=0x7f8b98004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:22.855 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:22.865+0000 7f8ba57fa700 1 -- 172.21.15.23:0/3700169541 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b90048ba0 con 0x7f8ba80f51a0 2021-11-19T08:39:23.074 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:23.089+0000 7f8badf1d700 1 -- 172.21.15.23:0/3700169541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f8ba80fed90 con 0x7f8ba80f51a0 2021-11-19T08:39:23.079 INFO:teuthology.orchestra.run.smithi023.stdout:[client.0] 2021-11-19T08:39:23.079 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQC7YpdhD1ZnBRAACXHr+GqB0DU4SmFvg+qing== 2021-11-19T08:39:23.080 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:23.094+0000 7f8ba57fa700 1 -- 172.21.15.23:0/3700169541 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f8b9004c1c0 con 0x7f8ba80f51a0 2021-11-19T08:39:23.081 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:23.096+0000 7f8badf1d700 1 -- 172.21.15.23:0/3700169541 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8b9405c300 msgr2=0x7f8b9405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:23.082 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:23.096+0000 7f8badf1d700 1 --2- 172.21.15.23:0/3700169541 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8b9405c300 0x7f8b9405e7c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f8b9800b620 tx=0x7f8b98004d10).stop 2021-11-19T08:39:23.082 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:23.096+0000 7f8badf1d700 1 -- 172.21.15.23:0/3700169541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ba80f51a0 msgr2=0x7f8ba8107460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:23.082 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:23.096+0000 7f8badf1d700 1 --2- 172.21.15.23:0/3700169541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ba80f51a0 0x7f8ba8107460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f8b900093f0 tx=0x7f8b90006ec0).stop 2021-11-19T08:39:23.082 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:23.096+0000 7f8badf1d700 1 -- 172.21.15.23:0/3700169541 shutdown_connections 2021-11-19T08:39:23.083 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:23.096+0000 7f8badf1d700 1 --2- 172.21.15.23:0/3700169541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ba80f4800 0x7f8ba8106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:23.083 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:23.096+0000 7f8badf1d700 1 --2- 172.21.15.23:0/3700169541 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8b9405c300 0x7f8b9405e7c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:23.083 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:23.096+0000 7f8badf1d700 1 --2- 172.21.15.23:0/3700169541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ba80f51a0 0x7f8ba8107460 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:23.083 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:23.096+0000 7f8badf1d700 1 -- 172.21.15.23:0/3700169541 >> 172.21.15.23:0/3700169541 conn(0x7f8ba80f0320 msgr2=0x7f8ba80fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:23.084 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:23.096+0000 7f8badf1d700 1 -- 172.21.15.23:0/3700169541 shutdown_connections 2021-11-19T08:39:23.084 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:23.096+0000 7f8badf1d700 1 -- 172.21.15.23:0/3700169541 wait complete. 2021-11-19T08:39:23.594 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2021-11-19T08:39:23.595 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2021-11-19T08:39:23.595 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-11-19T08:39:23.636 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-11-19T08:39:23.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:23 smithi134 conmon[36363]: audit 2021-11-19T08:39:23.090460+0000 mon.smithi023 (mon.0) 428 : audit [INF] from='client.? 172.21.15.23:0/3700169541' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-19T08:39:23.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:23 smithi134 conmon[36363]: audit 2021-11-19T08:39:23.095267+0000 2021-11-19T08:39:23.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:23 smithi134 conmon[36363]: mon.smithi023 (mon.0) 429 : audit [INF] from='client.? 172.21.15.23:0/3700169541' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-19T08:39:23.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:23 smithi023 conmon[33490]: audit 2021-11-19T08:39:23.090460+0000 mon.smithi023 (mon.0) 428 : audit [INF] from='client.? 172.21.15.23:0/3700169541' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-19T08:39:23.964 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:23 smithi023 conmon[33490]: audit 2021-11-19T08:39:23.095267+0000 mon.smithi023 (mon.0) 429 : audit [INF] from='client.? 172.21.15.23:0/3700169541' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-19T08:39:24.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:24 smithi134 conmon[36363]: cluster 2021-11-19T08:39:22.814469+0000 mgr.smithi023.ehlpcl (mgr.14164) 148 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:24.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:24 smithi023 conmon[33490]: cluster 2021-11-19T08:39:22.814469 2021-11-19T08:39:24.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:24 smithi023 conmon[33490]: +0000 mgr.smithi023.ehlpcl (mgr.14164) 148 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:25.226 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.239+0000 7f9759846700 1 -- 172.21.15.134:0/1818078828 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f97540f7980 msgr2=0x7f97540f7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:25.226 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.239+0000 7f9759846700 1 --2- 172.21.15.134:0/1818078828 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f97540f7980 0x7f97540f7de0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9748004660 tx=0x7f9748009b30).stop 2021-11-19T08:39:25.227 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.240+0000 7f9759846700 1 -- 172.21.15.134:0/1818078828 shutdown_connections 2021-11-19T08:39:25.227 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.240+0000 7f9759846700 1 --2- 172.21.15.134:0/1818078828 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f97540f7980 0x7f97540f7de0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:25.227 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.240+0000 7f9759846700 1 --2- 172.21.15.134:0/1818078828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97540f66e0 0x7f97540f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:25.227 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.240+0000 7f9759846700 1 -- 172.21.15.134:0/1818078828 >> 172.21.15.134:0/1818078828 conn(0x7f97540f1e40 msgr2=0x7f97540f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:25.228 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.240+0000 7f9759846700 1 -- 172.21.15.134:0/1818078828 shutdown_connections 2021-11-19T08:39:25.228 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.240+0000 7f9759846700 1 -- 172.21.15.134:0/1818078828 wait complete. 2021-11-19T08:39:25.228 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.240+0000 7f9759846700 1 Processor -- start 2021-11-19T08:39:25.228 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.240+0000 7f9759846700 1 -- start start 2021-11-19T08:39:25.228 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.240+0000 7f9759846700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f97540f66e0 0x7f9754068d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:25.229 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.240+0000 7f9759846700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97540f7980 0x7f9754069210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:25.229 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.240+0000 7f9759846700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9754069810 con 0x7f97540f7980 2021-11-19T08:39:25.229 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.240+0000 7f9759846700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9754110b60 con 0x7f97540f66e0 2021-11-19T08:39:25.229 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.241+0000 7f9752ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f97540f66e0 0x7f9754068d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:25.230 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.241+0000 7f9752ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f97540f66e0 0x7f9754068d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47992/0 (socket says 172.21.15.134:47992) 2021-11-19T08:39:25.230 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.241+0000 7f9752ffd700 1 -- 172.21.15.134:0/3998064183 learned_addr learned my addr 172.21.15.134:0/3998064183 (peer_addr_for_me v2:172.21.15.134:0/0) 2021-11-19T08:39:25.230 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.241+0000 7f97527fc700 1 --2- 172.21.15.134:0/3998064183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97540f7980 0x7f9754069210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:25.230 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.241+0000 7f9752ffd700 1 -- 172.21.15.134:0/3998064183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97540f7980 msgr2=0x7f9754069210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:25.230 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.241+0000 7f9752ffd700 1 --2- 172.21.15.134:0/3998064183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97540f7980 0x7f9754069210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:25.231 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.241+0000 7f9752ffd700 1 -- 172.21.15.134:0/3998064183 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9748005040 con 0x7f97540f66e0 2021-11-19T08:39:25.231 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.241+0000 7f9752ffd700 1 --2- 172.21.15.134:0/3998064183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f97540f66e0 0x7f9754068d20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f9744002700 tx=0x7f9744009d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:25.231 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.241+0000 7f973b7fe700 1 -- 172.21.15.134:0/3998064183 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97440079c0 con 0x7f97540f66e0 2021-11-19T08:39:25.231 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.241+0000 7f9759846700 1 -- 172.21.15.134:0/3998064183 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9754110e40 con 0x7f97540f66e0 2021-11-19T08:39:25.232 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.241+0000 7f973b7fe700 1 -- 172.21.15.134:0/3998064183 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f974400f040 con 0x7f97540f66e0 2021-11-19T08:39:25.232 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.241+0000 7f973b7fe700 1 -- 172.21.15.134:0/3998064183 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9744014860 con 0x7f97540f66e0 2021-11-19T08:39:25.232 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.241+0000 7f9759846700 1 -- 172.21.15.134:0/3998064183 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9754111400 con 0x7f97540f66e0 2021-11-19T08:39:25.235 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.242+0000 7f9759846700 1 -- 172.21.15.134:0/3998064183 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97540f5ad0 con 0x7f97540f66e0 2021-11-19T08:39:25.235 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.242+0000 7f973b7fe700 1 -- 172.21.15.134:0/3998064183 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f974401b060 con 0x7f97540f66e0 2021-11-19T08:39:25.235 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.243+0000 7f973b7fe700 1 --2- 172.21.15.134:0/3998064183 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f973c05c2b0 0x7f973c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:25.235 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.243+0000 7f973b7fe700 1 -- 172.21.15.134:0/3998064183 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f97440759d0 con 0x7f97540f66e0 2021-11-19T08:39:25.236 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.243+0000 7f97527fc700 1 --2- 172.21.15.134:0/3998064183 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f973c05c2b0 0x7f973c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:25.236 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.243+0000 7f97527fc700 1 --2- 172.21.15.134:0/3998064183 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f973c05c2b0 0x7f973c05e770 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f9754068ab0 tx=0x7f9748009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:25.236 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.247+0000 7f973b7fe700 1 -- 172.21.15.134:0/3998064183 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f97440489d0 con 0x7f97540f66e0 2021-11-19T08:39:25.435 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.450+0000 7f9759846700 1 -- 172.21.15.134:0/3998064183 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f9754045ad0 con 0x7f97540f66e0 2021-11-19T08:39:25.442 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.457+0000 7f973b7fe700 1 -- 172.21.15.134:0/3998064183 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f974404bff0 con 0x7f97540f66e0 2021-11-19T08:39:25.442 INFO:teuthology.orchestra.run.smithi134.stdout:[client.1] 2021-11-19T08:39:25.442 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQC9Ypdh1kb4GhAAn54szOaJpxqwQCChWAeOuA== 2021-11-19T08:39:25.443 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.458+0000 7f9759846700 1 -- 172.21.15.134:0/3998064183 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f973c05c2b0 msgr2=0x7f973c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:25.444 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.458+0000 7f9759846700 1 --2- 172.21.15.134:0/3998064183 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f973c05c2b0 0x7f973c05e770 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f9754068ab0 tx=0x7f9748009b30).stop 2021-11-19T08:39:25.444 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.458+0000 7f9759846700 1 -- 172.21.15.134:0/3998064183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f97540f66e0 msgr2=0x7f9754068d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:25.444 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.458+0000 7f9759846700 1 --2- 172.21.15.134:0/3998064183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f97540f66e0 0x7f9754068d20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f9744002700 tx=0x7f9744009d80).stop 2021-11-19T08:39:25.444 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.458+0000 7f9759846700 1 -- 172.21.15.134:0/3998064183 shutdown_connections 2021-11-19T08:39:25.444 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.458+0000 7f9759846700 1 --2- 172.21.15.134:0/3998064183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f97540f66e0 0x7f9754068d20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:25.445 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.458+0000 7f9759846700 1 --2- 172.21.15.134:0/3998064183 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f973c05c2b0 0x7f973c05e770 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:25.445 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.458+0000 7f9759846700 1 --2- 172.21.15.134:0/3998064183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97540f7980 0x7f9754069210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:25.445 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.458+0000 7f9759846700 1 -- 172.21.15.134:0/3998064183 >> 172.21.15.134:0/3998064183 conn(0x7f97540f1e40 msgr2=0x7f97540fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:25.445 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.459+0000 7f9759846700 1 -- 172.21.15.134:0/3998064183 shutdown_connections 2021-11-19T08:39:25.446 INFO:teuthology.orchestra.run.smithi134.stderr:2021-11-19T08:39:25.459+0000 7f9759846700 1 -- 172.21.15.134:0/3998064183 wait complete. 2021-11-19T08:39:25.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:25 smithi134 conmon[36363]: audit 2021-11-19T08:39:25.451969+0000 mon.smithi134 (mon.1 2021-11-19T08:39:25.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:25 smithi134 conmon[36363]: ) 14 : audit [INF] from='client.? 172.21.15.134:0/3998064183' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-19T08:39:25.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:25 smithi134 conmon[36363]: audit 2021-11-19T08:39:25.452258+0000 mon.smithi023 (mon.0) 430 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-19T08:39:25.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:25 smithi134 conmon[36363]: audit 2021-11-19T08:39:25.457553+0000 mon.smithi023 (mon.0) 431 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-19T08:39:25.847 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2021-11-19T08:39:25.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2021-11-19T08:39:25.848 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-11-19T08:39:25.887 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-19T08:39:25.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:25 smithi023 conmon[33490]: audit 2021-11-19T08:39:25.451969+0000 mon.smithi134 (mon.1 2021-11-19T08:39:25.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:25 smithi023 conmon[33490]: ) 14 : audit [INF] from='client.? 172.21.15.134:0/3998064183' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-19T08:39:25.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:25 smithi023 conmon[33490]: audit 2021-11-19T08:39:25.452258+0000 mon.smithi023 (mon.0) 430 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-11-19T08:39:25.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:25 smithi023 conmon[33490]: audit 2021-11-19T08:39:25.457553+0000 mon.smithi023 (mon.0) 431 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-11-19T08:39:26.336 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:26.760 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:26 smithi023 conmon[33490]: cluster 2021-11-19T08:39:24.814944+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:39:26.760 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:26 smithi023 conmon[33490]: .14164) 149 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:26.761 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:26 smithi023 conmon[33490]: audit 2021-11-19T08:39:26.219728+0000 mon.smithi023 (mon.0) 432 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:26.761 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:26 smithi023 conmon[33490]: 2021-11-19T08:39:26.761 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:26 smithi023 conmon[33490]: audit 2021-11-19T08:39:26.223722+0000 mon.smithi023 (mon.0) 433 : audit 2021-11-19T08:39:26.761 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:26 smithi023 conmon[33490]: [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:26.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:26 smithi134 conmon[36363]: cluster 2021-11-19T08:39:24.814944+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:39:26.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:26 smithi134 conmon[36363]: ) 149 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:26.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:26 smithi134 conmon[36363]: audit 2021-11-19T08:39:26.219728+0000 mon.smithi023 (mon.0) 432 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:26.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:26 smithi134 conmon[36363]: 2021-11-19T08:39:26.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:26 smithi134 conmon[36363]: audit 2021-11-19T08:39:26.223722+0000 mon.smithi023 (mon. 2021-11-19T08:39:26.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:26 smithi134 conmon[36363]: 0) 433 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:39:27.499 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.511+0000 7f859b716700 1 -- 172.21.15.23:0/4040486085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85940f59b0 msgr2=0x7f85940f5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:27.499 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.511+0000 7f859b716700 1 --2- 172.21.15.23:0/4040486085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85940f59b0 0x7f85940f5e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f8588004660 tx=0x7f8588009b30).stop 2021-11-19T08:39:27.499 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.511+0000 7f859b716700 1 -- 172.21.15.23:0/4040486085 shutdown_connections 2021-11-19T08:39:27.500 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.511+0000 7f859b716700 1 --2- 172.21.15.23:0/4040486085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85940f59b0 0x7f85940f5e30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:27.500 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.511+0000 7f859b716700 1 --2- 172.21.15.23:0/4040486085 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85940f6cb0 0x7f85940f70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:27.500 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.511+0000 7f859b716700 1 -- 172.21.15.23:0/4040486085 >> 172.21.15.23:0/4040486085 conn(0x7f85940f05e0 msgr2=0x7f85940f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:27.500 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.511+0000 7f859b716700 1 -- 172.21.15.23:0/4040486085 shutdown_connections 2021-11-19T08:39:27.501 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.511+0000 7f859b716700 1 -- 172.21.15.23:0/4040486085 wait complete. 2021-11-19T08:39:27.501 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.512+0000 7f859b716700 1 Processor -- start 2021-11-19T08:39:27.501 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.512+0000 7f859b716700 1 -- start start 2021-11-19T08:39:27.501 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.512+0000 7f859b716700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85940f59b0 0x7f8594105ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:27.502 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.512+0000 7f859b716700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85940f6cb0 0x7f85941064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:27.502 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.512+0000 7f859b716700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8594106a90 con 0x7f85940f59b0 2021-11-19T08:39:27.502 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.512+0000 7f859b716700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8594106bd0 con 0x7f85940f6cb0 2021-11-19T08:39:27.502 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.512+0000 7f85994b2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85940f59b0 0x7f8594105ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:27.502 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.512+0000 7f85994b2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85940f59b0 0x7f8594105ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46648/0 (socket says 172.21.15.23:46648) 2021-11-19T08:39:27.503 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.512+0000 7f85994b2700 1 -- 172.21.15.23:0/1441618218 learned_addr learned my addr 172.21.15.23:0/1441618218 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:27.503 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.512+0000 7f8598cb1700 1 --2- 172.21.15.23:0/1441618218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85940f6cb0 0x7f85941064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:27.503 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.513+0000 7f85994b2700 1 -- 172.21.15.23:0/1441618218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85940f6cb0 msgr2=0x7f85941064e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:27.503 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.513+0000 7f85994b2700 1 --2- 172.21.15.23:0/1441618218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85940f6cb0 0x7f85941064e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:27.504 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.513+0000 7f85994b2700 1 -- 172.21.15.23:0/1441618218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8588005040 con 0x7f85940f59b0 2021-11-19T08:39:27.504 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.513+0000 7f8598cb1700 1 --2- 172.21.15.23:0/1441618218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85940f6cb0 0x7f85941064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:39:27.504 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.513+0000 7f85994b2700 1 --2- 172.21.15.23:0/1441618218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85940f59b0 0x7f8594105ff0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f858c00f910 tx=0x7f858c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:27.504 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.513+0000 7f8591ffb700 1 -- 172.21.15.23:0/1441618218 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f858c009500 con 0x7f85940f59b0 2021-11-19T08:39:27.505 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.513+0000 7f8591ffb700 1 -- 172.21.15.23:0/1441618218 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f858c009660 con 0x7f85940f59b0 2021-11-19T08:39:27.505 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.513+0000 7f859b716700 1 -- 172.21.15.23:0/1441618218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8594107570 con 0x7f85940f59b0 2021-11-19T08:39:27.505 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.513+0000 7f8591ffb700 1 -- 172.21.15.23:0/1441618218 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f858c00de20 con 0x7f85940f59b0 2021-11-19T08:39:27.505 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.513+0000 7f859b716700 1 -- 172.21.15.23:0/1441618218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8594107b30 con 0x7f85940f59b0 2021-11-19T08:39:27.506 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.514+0000 7f859b716700 1 -- 172.21.15.23:0/1441618218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8594045ad0 con 0x7f85940f59b0 2021-11-19T08:39:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.514+0000 7f8591ffb700 1 -- 172.21.15.23:0/1441618218 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f858c00e040 con 0x7f85940f59b0 2021-11-19T08:39:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.515+0000 7f8591ffb700 1 --2- 172.21.15.23:0/1441618218 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f857c05c2a0 0x7f857c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.515+0000 7f8591ffb700 1 -- 172.21.15.23:0/1441618218 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f858c078220 con 0x7f85940f59b0 2021-11-19T08:39:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.515+0000 7f8598cb1700 1 --2- 172.21.15.23:0/1441618218 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f857c05c2a0 0x7f857c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.515+0000 7f8598cb1700 1 --2- 172.21.15.23:0/1441618218 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f857c05c2a0 0x7f857c05e760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f8588004e30 tx=0x7f8588009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.519+0000 7f8591ffb700 1 -- 172.21.15.23:0/1441618218 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f858c04b240 con 0x7f85940f59b0 2021-11-19T08:39:27.663 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.678+0000 7f859b716700 1 -- 172.21.15.23:0/1441618218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85941081c0 con 0x7f85940f59b0 2021-11-19T08:39:27.664 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.679+0000 7f8591ffb700 1 -- 172.21.15.23:0/1441618218 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+10918 (secure 0 0 0) 0x7f858c04e860 con 0x7f85940f59b0 2021-11-19T08:39:27.665 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:39:27.665 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":36,"fsid":"908b41ee-4913-11ec-8c2c-001a4aab830c","created":"2021-11-19T08:35:14.952900+0000","modified":"2021-11-19T08:39:21.519777+0000","last_up_change":"2021-11-19T08:39:20.513380+0000","last_in_change":"2021-11-19T08:39:05.162873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-19T08:38:42.679405+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"55d41e72-4389-4078-ab49-915578ae7162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6803","nonce":2382765305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6805","nonce":2382765305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6809","nonce":2382765305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6807","nonce":2382765305}]},"public_addr":"172.21.15.23:6803/2382765305","cluster_addr":"172.21.15.23:6805/2382765305","heartbeat_back_addr":"172.21.15.23:6809/2382765305","heartbeat_front_addr":"172.21.15.23:6807/2382765305","state":["exists","up"]},{"osd":1,"uuid":"5b7aea87-24b7-43e6-9ade-8fef0e9e6dd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6811","nonce":1051412153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6813","nonce":1051412153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6817","nonce":1051412153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6815","nonce":1051412153}]},"public_addr":"172.21.15.23:6811/1051412153","cluster_addr":"172.21.15.23:6813/1051412153","heartbeat_back_addr":"172.21.15.23:6817/1051412153","heartbeat_front_addr":"172.21.15.23:6815/1051412153","state":["exists","up"]},{"osd":2,"uuid":"e21cf6ec-4aa2-43cc-934a-bbd8d6e6694c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6819","nonce":954281054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6821","nonce":954281054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6825","nonce":954281054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6823","nonce":954281054}]},"public_addr":"172.21.15.23:6819/954281054","cluster_addr":"172.21.15.23:6821/954281054","heartbeat_back_addr":"172.21.15.23:6825/954281054","heartbeat_front_addr":"172.21.15.23:6823/954281054","state":["exists","up"]},{"osd":3,"uuid":"276cb410-5d23-4cdf-a090-e777ebe4c82f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6801","nonce":2145743581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6803","nonce":2145743581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6807","nonce":2145743581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6805","nonce":2145743581}]},"public_addr":"172.21.15.134:6801/2145743581","cluster_addr":"172.21.15.134:6803/2145743581","heartbeat_back_addr":"172.21.15.134:6807/2145743581","heartbeat_front_addr":"172.21.15.134:6805/2145743581","state":["exists","up"]},{"osd":4,"uuid":"015b6da1-35ed-43f4-84c9-337d2b0e79bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6809","nonce":1867765886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6811","nonce":1867765886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6815","nonce":1867765886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6813","nonce":1867765886}]},"public_addr":"172.21.15.134:6809/1867765886","cluster_addr":"172.21.15.134:6811/1867765886","heartbeat_back_addr":"172.21.15.134:6815/1867765886","heartbeat_front_addr":"172.21.15.134:6813/1867765886","state":["exists","up"]},{"osd":5,"uuid":"86ec41d3-38df-4ee0-a320-402055f7895b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6817","nonce":3784199400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6819","nonce":3784199400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6823","nonce":3784199400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6821","nonce":3784199400}]},"public_addr":"172.21.15.134:6817/3784199400","cluster_addr":"172.21.15.134:6819/3784199400","heartbeat_back_addr":"172.21.15.134:6823/3784199400","heartbeat_front_addr":"172.21.15.134:6821/3784199400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:37:52.252795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:12.408238+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:31.280017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:47.181003+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:39:03.121900+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:39:19.383582+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/192042719":"2021-11-20T08:36:18.730761+0000","172.21.15.23:0/4146945986":"2021-11-20T08:36:18.730761+0000","172.21.15.23:6801/158654977":"2021-11-20T08:36:18.730761+0000","172.21.15.23:6800/158654977":"2021-11-20T08:36:18.730761+0000","172.21.15.23:0/992169182":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/1001097146":"2021-11-20T08:36:02.243722+0000","172.21.15.23:6800/1352671878":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/3039325239":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6801/1352671878":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/4131282839":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6800/945698991":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6801/945698991":"2021-11-20T08:35:42.544921+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-19T08:39:27.666 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.681+0000 7f859b716700 1 -- 172.21.15.23:0/1441618218 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f857c05c2a0 msgr2=0x7f857c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:27.666 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.681+0000 7f859b716700 1 --2- 172.21.15.23:0/1441618218 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f857c05c2a0 0x7f857c05e760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f8588004e30 tx=0x7f8588009b30).stop 2021-11-19T08:39:27.667 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.681+0000 7f859b716700 1 -- 172.21.15.23:0/1441618218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85940f59b0 msgr2=0x7f8594105ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:27.667 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.681+0000 7f859b716700 1 --2- 172.21.15.23:0/1441618218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85940f59b0 0x7f8594105ff0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f858c00f910 tx=0x7f858c009d80).stop 2021-11-19T08:39:27.667 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.681+0000 7f859b716700 1 -- 172.21.15.23:0/1441618218 shutdown_connections 2021-11-19T08:39:27.667 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.681+0000 7f859b716700 1 --2- 172.21.15.23:0/1441618218 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f857c05c2a0 0x7f857c05e760 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:27.668 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.681+0000 7f859b716700 1 --2- 172.21.15.23:0/1441618218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85940f59b0 0x7f8594105ff0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:27.668 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.681+0000 7f859b716700 1 --2- 172.21.15.23:0/1441618218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f85940f6cb0 0x7f85941064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:27.668 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.681+0000 7f859b716700 1 -- 172.21.15.23:0/1441618218 >> 172.21.15.23:0/1441618218 conn(0x7f85940f05e0 msgr2=0x7f85940fa410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:27.668 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.681+0000 7f859b716700 1 -- 172.21.15.23:0/1441618218 shutdown_connections 2021-11-19T08:39:27.668 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:27.681+0000 7f859b716700 1 -- 172.21.15.23:0/1441618218 wait complete. 2021-11-19T08:39:28.051 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-19T08:38:42.679405+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2021-11-19T08:39:28.052 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph osd pool get device_health_metrics pg_num 2021-11-19T08:39:28.497 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:28.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:28 smithi134 conmon[36363]: cluster 2021-11-19T08:39:26.815254+0000 mgr.smithi023.ehlpcl (mgr.14164) 150 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:28.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:28 smithi134 conmon[36363]: audit 2021-11-19T08:39:27.679977+0000 mon.smithi023 (mon.0) 434 : audit [DBG] from='client.? 172.21.15.23:0/1441618218' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-19T08:39:28.829 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:28 smithi023 conmon[33490]: cluster 2021-11-19T08:39:26.815254+0000 mgr.smithi023.ehlpcl (mgr.14164) 150 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:28.830 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:28 smithi023 conmon[33490]: audit 2021-11-19T08:39:27.679977+0000 mon.smithi023 (mon.0) 434 : audit [DBG] from='client.? 172.21.15.23:0/1441618218' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-19T08:39:29.578 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.592+0000 7f2b3693c700 1 -- 172.21.15.23:0/2666337232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b300f7a70 msgr2=0x7f2b300f7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:29.578 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.592+0000 7f2b3693c700 1 --2- 172.21.15.23:0/2666337232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b300f7a70 0x7f2b300f7ed0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f2b24004660 tx=0x7f2b24009b30).stop 2021-11-19T08:39:29.578 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.592+0000 7f2b3693c700 1 -- 172.21.15.23:0/2666337232 shutdown_connections 2021-11-19T08:39:29.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.592+0000 7f2b3693c700 1 --2- 172.21.15.23:0/2666337232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b300f7a70 0x7f2b300f7ed0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:29.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.592+0000 7f2b3693c700 1 --2- 172.21.15.23:0/2666337232 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b300f67d0 0x7f2b300f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:29.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.592+0000 7f2b3693c700 1 -- 172.21.15.23:0/2666337232 >> 172.21.15.23:0/2666337232 conn(0x7f2b300f1f50 msgr2=0x7f2b300f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.592+0000 7f2b3693c700 1 -- 172.21.15.23:0/2666337232 shutdown_connections 2021-11-19T08:39:29.581 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.592+0000 7f2b3693c700 1 -- 172.21.15.23:0/2666337232 wait complete. 2021-11-19T08:39:29.581 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.593+0000 7f2b3693c700 1 Processor -- start 2021-11-19T08:39:29.582 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.593+0000 7f2b3693c700 1 -- start start 2021-11-19T08:39:29.590 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.593+0000 7f2b3693c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b300f67d0 0x7f2b301070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:29.591 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.593+0000 7f2b3693c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b300f7a70 0x7f2b301075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:29.592 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.593+0000 7f2b3693c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b30107b50 con 0x7f2b300f67d0 2021-11-19T08:39:29.594 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.593+0000 7f2b3693c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b30107c90 con 0x7f2b300f7a70 2021-11-19T08:39:29.595 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.593+0000 7f2b2ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b300f67d0 0x7f2b301070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:29.595 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.593+0000 7f2b2ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b300f67d0 0x7f2b301070b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46658/0 (socket says 172.21.15.23:46658) 2021-11-19T08:39:29.596 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.593+0000 7f2b2ffff700 1 -- 172.21.15.23:0/236956708 learned_addr learned my addr 172.21.15.23:0/236956708 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:29.596 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.593+0000 7f2b2f7fe700 1 --2- 172.21.15.23:0/236956708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b300f7a70 0x7f2b301075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:29.597 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.593+0000 7f2b2ffff700 1 -- 172.21.15.23:0/236956708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b300f7a70 msgr2=0x7f2b301075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:29.597 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.594+0000 7f2b2ffff700 1 --2- 172.21.15.23:0/236956708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b300f7a70 0x7f2b301075a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:29.597 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.594+0000 7f2b2ffff700 1 -- 172.21.15.23:0/236956708 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b24005040 con 0x7f2b300f67d0 2021-11-19T08:39:29.598 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.594+0000 7f2b2f7fe700 1 --2- 172.21.15.23:0/236956708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b300f7a70 0x7f2b301075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:39:29.598 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.594+0000 7f2b2ffff700 1 --2- 172.21.15.23:0/236956708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b300f67d0 0x7f2b301070b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f2b2000c790 tx=0x7f2b20007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:29.599 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.594+0000 7f2b2cff9700 1 -- 172.21.15.23:0/236956708 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b2000e680 con 0x7f2b300f67d0 2021-11-19T08:39:29.600 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.594+0000 7f2b3693c700 1 -- 172.21.15.23:0/236956708 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b30108630 con 0x7f2b300f67d0 2021-11-19T08:39:29.600 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.594+0000 7f2b2cff9700 1 -- 172.21.15.23:0/236956708 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b2000ec90 con 0x7f2b300f67d0 2021-11-19T08:39:29.601 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.594+0000 7f2b2cff9700 1 -- 172.21.15.23:0/236956708 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b20012cb0 con 0x7f2b300f67d0 2021-11-19T08:39:29.601 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.594+0000 7f2b3693c700 1 -- 172.21.15.23:0/236956708 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b30108bf0 con 0x7f2b300f67d0 2021-11-19T08:39:29.602 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.595+0000 7f2b3693c700 1 -- 172.21.15.23:0/236956708 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b30045ad0 con 0x7f2b300f67d0 2021-11-19T08:39:29.602 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.595+0000 7f2b2cff9700 1 -- 172.21.15.23:0/236956708 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2b20010070 con 0x7f2b300f67d0 2021-11-19T08:39:29.603 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.596+0000 7f2b2cff9700 1 --2- 172.21.15.23:0/236956708 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2b1805c2a0 0x7f2b1805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:29.603 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.596+0000 7f2b2cff9700 1 -- 172.21.15.23:0/236956708 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f2b20075c90 con 0x7f2b300f67d0 2021-11-19T08:39:29.604 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.596+0000 7f2b2f7fe700 1 --2- 172.21.15.23:0/236956708 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2b1805c2a0 0x7f2b1805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:29.605 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.596+0000 7f2b2f7fe700 1 --2- 172.21.15.23:0/236956708 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2b1805c2a0 0x7f2b1805e760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f2b24009d60 tx=0x7f2b24009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:29.605 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.600+0000 7f2b2cff9700 1 -- 172.21.15.23:0/236956708 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b20048c90 con 0x7f2b300f67d0 2021-11-19T08:39:29.746 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.760+0000 7f2b3693c700 1 -- 172.21.15.23:0/236956708 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f2b300ff060 con 0x7f2b300f67d0 2021-11-19T08:39:29.746 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.760+0000 7f2b2cff9700 1 -- 172.21.15.23:0/236956708 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v36) v1 ==== 110+0+10 (secure 0 0 0) 0x7f2b2004c2b0 con 0x7f2b300f67d0 2021-11-19T08:39:29.747 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2021-11-19T08:39:29.748 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.762+0000 7f2b3693c700 1 -- 172.21.15.23:0/236956708 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2b1805c2a0 msgr2=0x7f2b1805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:29.749 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.762+0000 7f2b3693c700 1 --2- 172.21.15.23:0/236956708 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2b1805c2a0 0x7f2b1805e760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f2b24009d60 tx=0x7f2b24009b30).stop 2021-11-19T08:39:29.750 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.762+0000 7f2b3693c700 1 -- 172.21.15.23:0/236956708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b300f67d0 msgr2=0x7f2b301070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:29.750 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.762+0000 7f2b3693c700 1 --2- 172.21.15.23:0/236956708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b300f67d0 0x7f2b301070b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f2b2000c790 tx=0x7f2b20007d90).stop 2021-11-19T08:39:29.751 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.762+0000 7f2b3693c700 1 -- 172.21.15.23:0/236956708 shutdown_connections 2021-11-19T08:39:29.751 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.762+0000 7f2b3693c700 1 --2- 172.21.15.23:0/236956708 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2b1805c2a0 0x7f2b1805e760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:29.751 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.762+0000 7f2b3693c700 1 --2- 172.21.15.23:0/236956708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b300f67d0 0x7f2b301070b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:29.751 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.762+0000 7f2b3693c700 1 --2- 172.21.15.23:0/236956708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b300f7a70 0x7f2b301075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:29.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.762+0000 7f2b3693c700 1 -- 172.21.15.23:0/236956708 >> 172.21.15.23:0/236956708 conn(0x7f2b300f1f50 msgr2=0x7f2b300fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:29.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.762+0000 7f2b3693c700 1 -- 172.21.15.23:0/236956708 shutdown_connections 2021-11-19T08:39:29.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:29.762+0000 7f2b3693c700 1 -- 172.21.15.23:0/236956708 wait complete. 2021-11-19T08:39:30.113 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-11-19T08:39:30.114 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2021-11-19T08:39:30.115 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph mgr dump --format=json 2021-11-19T08:39:30.557 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:30.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:30 smithi134 conmon[36363]: cluster 2021-11-19T08:39:28.815622+0000 mgr.smithi023.ehlpcl (mgr.14164) 151 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:30.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:30 smithi134 conmon[36363]: 2021-11-19T08:39:30.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:30 smithi134 conmon[36363]: audit 2021-11-19T08:39:29.761632+0000 mon.smithi023 (mon.0) 435 : audit [DBG] from='client.? 172.21.15.23:0/236956708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-19T08:39:30.803 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:30 smithi023 conmon[33490]: cluster 2021-11-19T08:39:28.815622+0000 mgr.smithi023.ehlpcl (mgr.14164) 151 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:30.803 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:30 smithi023 conmon[33490]: audit 2021-11-19T08:39:29.761632+0000 mon.smithi023 (mon.0) 435 : audit [DBG] from='client.? 172.21.15.23:0/236956708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-19T08:39:31.705 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.718+0000 7f8671677700 1 -- 172.21.15.23:0/3346753278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c0f4aa0 msgr2=0x7f866c0f4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:31.706 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.718+0000 7f8671677700 1 --2- 172.21.15.23:0/3346753278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c0f4aa0 0x7f866c0f4ec0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f865c004660 tx=0x7f865c009b30).stop 2021-11-19T08:39:31.706 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.718+0000 7f8671677700 1 -- 172.21.15.23:0/3346753278 shutdown_connections 2021-11-19T08:39:31.706 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.718+0000 7f8671677700 1 --2- 172.21.15.23:0/3346753278 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f866c0f5440 0x7f866c0f7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-19T08:39:31.707 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.718+0000 7f8671677700 1 --2- 172.21.15.23:0/3346753278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c0f4aa0 0x7f866c0f4ec0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:31.707 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.718+0000 7f8671677700 1 -- 172.21.15.23:0/3346753278 >> 172.21.15.23:0/3346753278 conn(0x7f866c0f05e0 msgr2=0x7f866c0f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:31.708 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.718+0000 7f8671677700 1 -- 172.21.15.23:0/3346753278 shutdown_connections 2021-11-19T08:39:31.708 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.718+0000 7f8671677700 1 -- 172.21.15.23:0/3346753278 wait complete. 2021-11-19T08:39:31.708 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.719+0000 7f8671677700 1 Processor -- start 2021-11-19T08:39:31.708 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.719+0000 7f8671677700 1 -- start start 2021-11-19T08:39:31.709 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.719+0000 7f8671677700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f866c0f4aa0 0x7f866c107160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:31.709 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.719+0000 7f8671677700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c0f5440 0x7f866c107650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:31.709 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.719+0000 7f8671677700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866c107c90 con 0x7f866c0f5440 2021-11-19T08:39:31.709 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.719+0000 7f8671677700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866c108540 con 0x7f866c0f4aa0 2021-11-19T08:39:31.710 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.719+0000 7f866a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c0f5440 0x7f866c107650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:31.710 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.719+0000 7f866a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c0f5440 0x7f866c107650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46668/0 (socket says 172.21.15.23:46668) 2021-11-19T08:39:31.710 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.719+0000 7f866a7fc700 1 -- 172.21.15.23:0/491071567 learned_addr learned my addr 172.21.15.23:0/491071567 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:31.710 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.720+0000 7f866affd700 1 --2- 172.21.15.23:0/491071567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f866c0f4aa0 0x7f866c107160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:31.711 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.720+0000 7f866a7fc700 1 -- 172.21.15.23:0/491071567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f866c0f4aa0 msgr2=0x7f866c107160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:31.711 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.720+0000 7f866a7fc700 1 --2- 172.21.15.23:0/491071567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f866c0f4aa0 0x7f866c107160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:31.711 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.720+0000 7f866a7fc700 1 -- 172.21.15.23:0/491071567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c005040 con 0x7f866c0f5440 2021-11-19T08:39:31.711 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.720+0000 7f866a7fc700 1 --2- 172.21.15.23:0/491071567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c0f5440 0x7f866c107650 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f86540093f0 tx=0x7f8654006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:31.711 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.720+0000 7f86637fe700 1 -- 172.21.15.23:0/491071567 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8654009ea0 con 0x7f866c0f5440 2021-11-19T08:39:31.712 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.720+0000 7f8671677700 1 -- 172.21.15.23:0/491071567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866c108820 con 0x7f866c0f5440 2021-11-19T08:39:31.712 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.720+0000 7f86637fe700 1 -- 172.21.15.23:0/491071567 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8654013af0 con 0x7f866c0f5440 2021-11-19T08:39:31.712 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.720+0000 7f8671677700 1 -- 172.21.15.23:0/491071567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866c108de0 con 0x7f866c0f5440 2021-11-19T08:39:31.712 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.720+0000 7f86637fe700 1 -- 172.21.15.23:0/491071567 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f865400dad0 con 0x7f866c0f5440 2021-11-19T08:39:31.713 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.721+0000 7f8671677700 1 -- 172.21.15.23:0/491071567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f866c059100 con 0x7f866c0f5440 2021-11-19T08:39:31.714 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.721+0000 7f86637fe700 1 -- 172.21.15.23:0/491071567 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f865401a060 con 0x7f866c0f5440 2021-11-19T08:39:31.715 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.722+0000 7f86637fe700 1 --2- 172.21.15.23:0/491071567 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f865805c2a0 0x7f865805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:31.715 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.722+0000 7f86637fe700 1 -- 172.21.15.23:0/491071567 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8654075bb0 con 0x7f866c0f5440 2021-11-19T08:39:31.715 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.722+0000 7f866affd700 1 --2- 172.21.15.23:0/491071567 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f865805c2a0 0x7f865805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:31.716 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.722+0000 7f866affd700 1 --2- 172.21.15.23:0/491071567 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f865805c2a0 0x7f865805e760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f865c004e30 tx=0x7f865c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:31.716 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.726+0000 7f86637fe700 1 -- 172.21.15.23:0/491071567 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f865407c080 con 0x7f866c0f5440 2021-11-19T08:39:31.907 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.922+0000 7f8671677700 1 -- 172.21.15.23:0/491071567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f866c0ff030 con 0x7f866c0f5440 2021-11-19T08:39:31.910 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.925+0000 7f86637fe700 1 -- 172.21.15.23:0/491071567 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v18) v1 ==== 74+0+143500 (secure 0 0 0) 0x7f8654048bb0 con 0x7f866c0f5440 2021-11-19T08:39:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:39:31.915 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.927+0000 7f8671677700 1 -- 172.21.15.23:0/491071567 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f865805c2a0 msgr2=0x7f865805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:31.915 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.927+0000 7f8671677700 1 --2- 172.21.15.23:0/491071567 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f865805c2a0 0x7f865805e760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f865c004e30 tx=0x7f865c009b30).stop 2021-11-19T08:39:31.915 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.927+0000 7f8671677700 1 -- 172.21.15.23:0/491071567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c0f5440 msgr2=0x7f866c107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:31.915 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.927+0000 7f8671677700 1 --2- 172.21.15.23:0/491071567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c0f5440 0x7f866c107650 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f86540093f0 tx=0x7f8654006ec0).stop 2021-11-19T08:39:31.916 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.927+0000 7f8671677700 1 -- 172.21.15.23:0/491071567 shutdown_connections 2021-11-19T08:39:31.916 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.927+0000 7f8671677700 1 --2- 172.21.15.23:0/491071567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f866c0f4aa0 0x7f866c107160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:31.916 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.927+0000 7f8671677700 1 --2- 172.21.15.23:0/491071567 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f865805c2a0 0x7f865805e760 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:31.916 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.927+0000 7f8671677700 1 --2- 172.21.15.23:0/491071567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f866c0f5440 0x7f866c107650 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:31.916 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.927+0000 7f8671677700 1 -- 172.21.15.23:0/491071567 >> 172.21.15.23:0/491071567 conn(0x7f866c0f05e0 msgr2=0x7f866c0fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:31.917 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.927+0000 7f8671677700 1 -- 172.21.15.23:0/491071567 shutdown_connections 2021-11-19T08:39:31.917 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:31.927+0000 7f8671677700 1 -- 172.21.15.23:0/491071567 wait complete. 2021-11-19T08:39:32.275 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":18,"active_gid":14164,"active_name":"smithi023.ehlpcl","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":3049298848},{"type":"v1","addr":"172.21.15.23:6801","nonce":3049298848}]},"active_addr":"172.21.15.23:6801/3049298848","active_change":"2021-11-19T08:36:18.731258+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":14198,"name":"smithi134.vitoqy","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://smithi023.front.sepia.ceph.com:8443/","prometheus":"http://smithi023.front.sepia.ceph.com:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":4,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":2088182372}]},{"addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":3051368353}]}]}} 2021-11-19T08:39:32.281 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2021-11-19T08:39:32.281 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2021-11-19T08:39:32.282 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-19T08:39:32.725 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:32.859 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:32 smithi023 conmon[33490]: cluster 2021-11-19T08:39:30.815909+0000 mgr.smithi023.ehlpcl (mgr.14164) 152 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:32.859 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:32 smithi023 conmon[33490]: audit 2021-11-19T08:39:31.923857+0000 mon.smithi023 (mon.0) 436 : audit [DBG] from='client.? 172.21.15.23:0/491071567' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-11-19T08:39:33.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:32 smithi134 conmon[36363]: cluster 2021-11-19T08:39:30.815909+0000 mgr.smithi023.ehlpcl (mgr.14164) 152 : 2021-11-19T08:39:33.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:32 smithi134 conmon[36363]: cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:33.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:32 smithi134 conmon[36363]: audit 2021-11-19T08:39:31.923857+0000 mon.smithi023 (mon.0) 436 : audit [DBG] from='client.? 172.21.15.23:0/491071567' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-11-19T08:39:34.023 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.035+0000 7fe96b59e700 1 -- 172.21.15.23:0/2282480214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c0f9790 msgr2=0x7fe96c0fbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:34.023 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.035+0000 7fe96b59e700 1 --2- 172.21.15.23:0/2282480214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c0f9790 0x7fe96c0fbbe0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fe954004660 tx=0x7fe954009b30).stop 2021-11-19T08:39:34.023 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.035+0000 7fe96b59e700 1 -- 172.21.15.23:0/2282480214 shutdown_connections 2021-11-19T08:39:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.035+0000 7fe96b59e700 1 --2- 172.21.15.23:0/2282480214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c0f9790 0x7fe96c0fbbe0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.035+0000 7fe96b59e700 1 --2- 172.21.15.23:0/2282480214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe96c0607d0 0x7fe96c0f92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-19T08:39:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.035+0000 7fe96b59e700 1 -- 172.21.15.23:0/2282480214 >> 172.21.15.23:0/2282480214 conn(0x7fe96c0f0ca0 msgr2=0x7fe96c0f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.035+0000 7fe96b59e700 1 -- 172.21.15.23:0/2282480214 shutdown_connections 2021-11-19T08:39:34.025 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.035+0000 7fe96b59e700 1 -- 172.21.15.23:0/2282480214 wait complete. 2021-11-19T08:39:34.025 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.036+0000 7fe96b59e700 1 Processor -- start 2021-11-19T08:39:34.025 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.036+0000 7fe96b59e700 1 -- start start 2021-11-19T08:39:34.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.036+0000 7fe96b59e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe96c0607d0 0x7fe96c106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:34.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.036+0000 7fe96b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c0f9790 0x7fe96c1073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:34.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.036+0000 7fe96b59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c107a10 con 0x7fe96c0f9790 2021-11-19T08:39:34.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.036+0000 7fe96b59e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c1082c0 con 0x7fe96c0607d0 2021-11-19T08:39:34.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.036+0000 7fe969d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c0f9790 0x7fe96c1073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:34.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.036+0000 7fe969d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c0f9790 0x7fe96c1073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46678/0 (socket says 172.21.15.23:46678) 2021-11-19T08:39:34.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.036+0000 7fe969d9b700 1 -- 172.21.15.23:0/2510791838 learned_addr learned my addr 172.21.15.23:0/2510791838 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:34.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.036+0000 7fe96a59c700 1 --2- 172.21.15.23:0/2510791838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe96c0607d0 0x7fe96c106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:34.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.036+0000 7fe969d9b700 1 -- 172.21.15.23:0/2510791838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe96c0607d0 msgr2=0x7fe96c106ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:34.029 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.036+0000 7fe969d9b700 1 --2- 172.21.15.23:0/2510791838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe96c0607d0 0x7fe96c106ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:34.029 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.037+0000 7fe969d9b700 1 -- 172.21.15.23:0/2510791838 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe954005040 con 0x7fe96c0f9790 2021-11-19T08:39:34.029 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.037+0000 7fe96a59c700 1 --2- 172.21.15.23:0/2510791838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe96c0607d0 0x7fe96c106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:39:34.030 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.037+0000 7fe969d9b700 1 --2- 172.21.15.23:0/2510791838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c0f9790 0x7fe96c1073d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fe954009f50 tx=0x7fe954009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:34.030 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.037+0000 7fe962ffd700 1 -- 172.21.15.23:0/2510791838 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe954018070 con 0x7fe96c0f9790 2021-11-19T08:39:34.030 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.037+0000 7fe962ffd700 1 -- 172.21.15.23:0/2510791838 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe954013410 con 0x7fe96c0f9790 2021-11-19T08:39:34.030 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.037+0000 7fe96b59e700 1 -- 172.21.15.23:0/2510791838 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96c108540 con 0x7fe96c0f9790 2021-11-19T08:39:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.037+0000 7fe962ffd700 1 -- 172.21.15.23:0/2510791838 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe95401c620 con 0x7fe96c0f9790 2021-11-19T08:39:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.037+0000 7fe96b59e700 1 -- 172.21.15.23:0/2510791838 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96c108aa0 con 0x7fe96c0f9790 2021-11-19T08:39:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.038+0000 7fe96b59e700 1 -- 172.21.15.23:0/2510791838 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe96c059100 con 0x7fe96c0f9790 2021-11-19T08:39:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.038+0000 7fe962ffd700 1 -- 172.21.15.23:0/2510791838 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe95401d420 con 0x7fe96c0f9790 2021-11-19T08:39:34.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.038+0000 7fe962ffd700 1 --2- 172.21.15.23:0/2510791838 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe95805c2b0 0x7fe95805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:34.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.039+0000 7fe962ffd700 1 -- 172.21.15.23:0/2510791838 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fe9540779a0 con 0x7fe96c0f9790 2021-11-19T08:39:34.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.039+0000 7fe96a59c700 1 --2- 172.21.15.23:0/2510791838 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe95805c2b0 0x7fe95805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:34.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.039+0000 7fe96a59c700 1 --2- 172.21.15.23:0/2510791838 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe95805c2b0 0x7fe95805e770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fe95c006fd0 tx=0x7fe95c008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:34.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.043+0000 7fe962ffd700 1 -- 172.21.15.23:0/2510791838 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe95404aa20 con 0x7fe96c0f9790 2021-11-19T08:39:34.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.202+0000 7fe96b59e700 1 -- 172.21.15.23:0/2510791838 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe96c0008d0 con 0x7fe96c0f9790 2021-11-19T08:39:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.203+0000 7fe962ffd700 1 -- 172.21.15.23:0/2510791838 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+10918 (secure 0 0 0) 0x7fe95404e040 con 0x7fe96c0f9790 2021-11-19T08:39:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:39:34.189 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":36,"fsid":"908b41ee-4913-11ec-8c2c-001a4aab830c","created":"2021-11-19T08:35:14.952900+0000","modified":"2021-11-19T08:39:21.519777+0000","last_up_change":"2021-11-19T08:39:20.513380+0000","last_in_change":"2021-11-19T08:39:05.162873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-19T08:38:42.679405+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"55d41e72-4389-4078-ab49-915578ae7162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6803","nonce":2382765305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6805","nonce":2382765305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6809","nonce":2382765305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6807","nonce":2382765305}]},"public_addr":"172.21.15.23:6803/2382765305","cluster_addr":"172.21.15.23:6805/2382765305","heartbeat_back_addr":"172.21.15.23:6809/2382765305","heartbeat_front_addr":"172.21.15.23:6807/2382765305","state":["exists","up"]},{"osd":1,"uuid":"5b7aea87-24b7-43e6-9ade-8fef0e9e6dd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6811","nonce":1051412153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6813","nonce":1051412153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6817","nonce":1051412153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6815","nonce":1051412153}]},"public_addr":"172.21.15.23:6811/1051412153","cluster_addr":"172.21.15.23:6813/1051412153","heartbeat_back_addr":"172.21.15.23:6817/1051412153","heartbeat_front_addr":"172.21.15.23:6815/1051412153","state":["exists","up"]},{"osd":2,"uuid":"e21cf6ec-4aa2-43cc-934a-bbd8d6e6694c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6819","nonce":954281054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6821","nonce":954281054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6825","nonce":954281054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6823","nonce":954281054}]},"public_addr":"172.21.15.23:6819/954281054","cluster_addr":"172.21.15.23:6821/954281054","heartbeat_back_addr":"172.21.15.23:6825/954281054","heartbeat_front_addr":"172.21.15.23:6823/954281054","state":["exists","up"]},{"osd":3,"uuid":"276cb410-5d23-4cdf-a090-e777ebe4c82f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6801","nonce":2145743581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6803","nonce":2145743581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6807","nonce":2145743581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6805","nonce":2145743581}]},"public_addr":"172.21.15.134:6801/2145743581","cluster_addr":"172.21.15.134:6803/2145743581","heartbeat_back_addr":"172.21.15.134:6807/2145743581","heartbeat_front_addr":"172.21.15.134:6805/2145743581","state":["exists","up"]},{"osd":4,"uuid":"015b6da1-35ed-43f4-84c9-337d2b0e79bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6809","nonce":1867765886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6811","nonce":1867765886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6815","nonce":1867765886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6813","nonce":1867765886}]},"public_addr":"172.21.15.134:6809/1867765886","cluster_addr":"172.21.15.134:6811/1867765886","heartbeat_back_addr":"172.21.15.134:6815/1867765886","heartbeat_front_addr":"172.21.15.134:6813/1867765886","state":["exists","up"]},{"osd":5,"uuid":"86ec41d3-38df-4ee0-a320-402055f7895b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6817","nonce":3784199400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6819","nonce":3784199400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6823","nonce":3784199400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6821","nonce":3784199400}]},"public_addr":"172.21.15.134:6817/3784199400","cluster_addr":"172.21.15.134:6819/3784199400","heartbeat_back_addr":"172.21.15.134:6823/3784199400","heartbeat_front_addr":"172.21.15.134:6821/3784199400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:37:52.252795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:12.408238+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:31.280017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:47.181003+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:39:03.121900+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:39:19.383582+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/192042719":"2021-11-20T08:36:18.730761+0000","172.21.15.23:0/4146945986":"2021-11-20T08:36:18.730761+0000","172.21.15.23:6801/158654977":"2021-11-20T08:36:18.730761+0000","172.21.15.23:6800/158654977":"2021-11-20T08:36:18.730761+0000","172.21.15.23:0/992169182":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/1001097146":"2021-11-20T08:36:02.243722+0000","172.21.15.23:6800/1352671878":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/3039325239":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6801/1352671878":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/4131282839":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6800/945698991":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6801/945698991":"2021-11-20T08:35:42.544921+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-19T08:39:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.204+0000 7fe96b59e700 1 -- 172.21.15.23:0/2510791838 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe95805c2b0 msgr2=0x7fe95805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.204+0000 7fe96b59e700 1 --2- 172.21.15.23:0/2510791838 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe95805c2b0 0x7fe95805e770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fe95c006fd0 tx=0x7fe95c008040).stop 2021-11-19T08:39:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.204+0000 7fe96b59e700 1 -- 172.21.15.23:0/2510791838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c0f9790 msgr2=0x7fe96c1073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:34.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.204+0000 7fe96b59e700 1 --2- 172.21.15.23:0/2510791838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c0f9790 0x7fe96c1073d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fe954009f50 tx=0x7fe954009b30).stop 2021-11-19T08:39:34.192 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.204+0000 7fe96b59e700 1 -- 172.21.15.23:0/2510791838 shutdown_connections 2021-11-19T08:39:34.192 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.204+0000 7fe96b59e700 1 --2- 172.21.15.23:0/2510791838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe96c0607d0 0x7fe96c106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:34.192 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.204+0000 7fe96b59e700 1 --2- 172.21.15.23:0/2510791838 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe95805c2b0 0x7fe95805e770 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:34.192 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.204+0000 7fe96b59e700 1 --2- 172.21.15.23:0/2510791838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c0f9790 0x7fe96c1073d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:34.193 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.204+0000 7fe96b59e700 1 -- 172.21.15.23:0/2510791838 >> 172.21.15.23:0/2510791838 conn(0x7fe96c0f0ca0 msgr2=0x7fe96c0f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:34.193 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.204+0000 7fe96b59e700 1 -- 172.21.15.23:0/2510791838 shutdown_connections 2021-11-19T08:39:34.193 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:34.204+0000 7fe96b59e700 1 -- 172.21.15.23:0/2510791838 wait complete. 2021-11-19T08:39:34.541 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2021-11-19T08:39:34.542 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph osd dump --format=json 2021-11-19T08:39:34.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:34 smithi023 conmon[33490]: cluster 2021-11-19T08:39:32.816193+0000 2021-11-19T08:39:34.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:34 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 153 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:34.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:34 smithi023 conmon[33490]: audit 2021-11-19T08:39:34.203476+0000 mon.smithi023 (mon.0) 437 : audit [DBG] from='client.? 172.21.15.23:0/2510791838' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-19T08:39:34.987 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:35.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:34 smithi134 conmon[36363]: cluster 2021-11-19T08:39:32.816193+0000 mgr.smithi023.ehlpcl (mgr.14164) 153 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:35.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:34 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:39:35.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:34 smithi134 conmon[36363]: 11-19T08:39:34.203476+0000 mon.smithi023 (mon.0) 437 : audit [DBG] from='client.? 172.21.15.23:0/2510791838' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-19T08:39:36.199 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.211+0000 7f6ace356700 1 -- 172.21.15.23:0/1860244935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ac80f67c0 msgr2=0x7f6ac80f6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:36.199 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.211+0000 7f6ace356700 1 --2- 172.21.15.23:0/1860244935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ac80f67c0 0x7f6ac80f6c20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6ab8004660 tx=0x7f6ab8009b30).stop 2021-11-19T08:39:36.199 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.211+0000 7f6ace356700 1 -- 172.21.15.23:0/1860244935 shutdown_connections 2021-11-19T08:39:36.200 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.211+0000 7f6ace356700 1 --2- 172.21.15.23:0/1860244935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ac80f67c0 0x7f6ac80f6c20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:36.200 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.211+0000 7f6ace356700 1 --2- 172.21.15.23:0/1860244935 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ac80f5520 0x7f6ac80f5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:36.200 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.211+0000 7f6ace356700 1 -- 172.21.15.23:0/1860244935 >> 172.21.15.23:0/1860244935 conn(0x7f6ac80f0ca0 msgr2=0x7f6ac80f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:36.200 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.211+0000 7f6ace356700 1 -- 172.21.15.23:0/1860244935 shutdown_connections 2021-11-19T08:39:36.201 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.211+0000 7f6ace356700 1 -- 172.21.15.23:0/1860244935 wait complete. 2021-11-19T08:39:36.201 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.212+0000 7f6ace356700 1 Processor -- start 2021-11-19T08:39:36.201 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.212+0000 7f6ace356700 1 -- start start 2021-11-19T08:39:36.201 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.212+0000 7f6ace356700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ac80f5520 0x7f6ac80fd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:36.201 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.212+0000 7f6ace356700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ac80f67c0 0x7f6ac80fbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.212+0000 7f6ace356700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac80fc080 con 0x7f6ac80f5520 2021-11-19T08:39:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.212+0000 7f6ace356700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac80fc1c0 con 0x7f6ac80f67c0 2021-11-19T08:39:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.212+0000 7f6acd354700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ac80f5520 0x7f6ac80fd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.212+0000 7f6acd354700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ac80f5520 0x7f6ac80fd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46686/0 (socket says 172.21.15.23:46686) 2021-11-19T08:39:36.203 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.212+0000 7f6acd354700 1 -- 172.21.15.23:0/712166983 learned_addr learned my addr 172.21.15.23:0/712166983 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:36.203 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.213+0000 7f6accb53700 1 --2- 172.21.15.23:0/712166983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ac80f67c0 0x7f6ac80fbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:36.203 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.213+0000 7f6acd354700 1 -- 172.21.15.23:0/712166983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ac80f67c0 msgr2=0x7f6ac80fbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:36.203 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.213+0000 7f6acd354700 1 --2- 172.21.15.23:0/712166983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ac80f67c0 0x7f6ac80fbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:36.203 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.213+0000 7f6acd354700 1 -- 172.21.15.23:0/712166983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab8005040 con 0x7f6ac80f5520 2021-11-19T08:39:36.204 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.213+0000 7f6acd354700 1 --2- 172.21.15.23:0/712166983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ac80f5520 0x7f6ac80fd6a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f6ac400c790 tx=0x7f6ac4007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:36.204 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.213+0000 7f6abdffb700 1 -- 172.21.15.23:0/712166983 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ac400e680 con 0x7f6ac80f5520 2021-11-19T08:39:36.204 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.213+0000 7f6abdffb700 1 -- 172.21.15.23:0/712166983 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ac400ec90 con 0x7f6ac80f5520 2021-11-19T08:39:36.204 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.213+0000 7f6ace356700 1 -- 172.21.15.23:0/712166983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac80fc4a0 con 0x7f6ac80f5520 2021-11-19T08:39:36.205 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.213+0000 7f6abdffb700 1 -- 172.21.15.23:0/712166983 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ac4012cb0 con 0x7f6ac80f5520 2021-11-19T08:39:36.205 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.213+0000 7f6ace356700 1 -- 172.21.15.23:0/712166983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac810eef0 con 0x7f6ac80f5520 2021-11-19T08:39:36.205 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.214+0000 7f6ace356700 1 -- 172.21.15.23:0/712166983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac8045ad0 con 0x7f6ac80f5520 2021-11-19T08:39:36.206 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.214+0000 7f6abdffb700 1 -- 172.21.15.23:0/712166983 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6ac4010070 con 0x7f6ac80f5520 2021-11-19T08:39:36.206 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.215+0000 7f6abdffb700 1 --2- 172.21.15.23:0/712166983 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6ab405c2a0 0x7f6ab405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:36.207 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.215+0000 7f6abdffb700 1 -- 172.21.15.23:0/712166983 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f6ac4075bc0 con 0x7f6ac80f5520 2021-11-19T08:39:36.207 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.215+0000 7f6accb53700 1 --2- 172.21.15.23:0/712166983 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6ab405c2a0 0x7f6ab405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:36.207 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.215+0000 7f6accb53700 1 --2- 172.21.15.23:0/712166983 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6ab405c2a0 0x7f6ab405e760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f6ab8004e30 tx=0x7f6ab8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:36.208 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.219+0000 7f6abdffb700 1 -- 172.21.15.23:0/712166983 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ac4048c40 con 0x7f6ac80f5520 2021-11-19T08:39:36.362 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.377+0000 7f6ace356700 1 -- 172.21.15.23:0/712166983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ac8027a00 con 0x7f6ac80f5520 2021-11-19T08:39:36.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.378+0000 7f6abdffb700 1 -- 172.21.15.23:0/712166983 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+10918 (secure 0 0 0) 0x7f6ac404c260 con 0x7f6ac80f5520 2021-11-19T08:39:36.364 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:39:36.364 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":36,"fsid":"908b41ee-4913-11ec-8c2c-001a4aab830c","created":"2021-11-19T08:35:14.952900+0000","modified":"2021-11-19T08:39:21.519777+0000","last_up_change":"2021-11-19T08:39:20.513380+0000","last_in_change":"2021-11-19T08:39:05.162873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-19T08:38:42.679405+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"55d41e72-4389-4078-ab49-915578ae7162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6803","nonce":2382765305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6805","nonce":2382765305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6809","nonce":2382765305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6807","nonce":2382765305}]},"public_addr":"172.21.15.23:6803/2382765305","cluster_addr":"172.21.15.23:6805/2382765305","heartbeat_back_addr":"172.21.15.23:6809/2382765305","heartbeat_front_addr":"172.21.15.23:6807/2382765305","state":["exists","up"]},{"osd":1,"uuid":"5b7aea87-24b7-43e6-9ade-8fef0e9e6dd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6811","nonce":1051412153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6813","nonce":1051412153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6817","nonce":1051412153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6815","nonce":1051412153}]},"public_addr":"172.21.15.23:6811/1051412153","cluster_addr":"172.21.15.23:6813/1051412153","heartbeat_back_addr":"172.21.15.23:6817/1051412153","heartbeat_front_addr":"172.21.15.23:6815/1051412153","state":["exists","up"]},{"osd":2,"uuid":"e21cf6ec-4aa2-43cc-934a-bbd8d6e6694c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6819","nonce":954281054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6821","nonce":954281054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6825","nonce":954281054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6823","nonce":954281054}]},"public_addr":"172.21.15.23:6819/954281054","cluster_addr":"172.21.15.23:6821/954281054","heartbeat_back_addr":"172.21.15.23:6825/954281054","heartbeat_front_addr":"172.21.15.23:6823/954281054","state":["exists","up"]},{"osd":3,"uuid":"276cb410-5d23-4cdf-a090-e777ebe4c82f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6801","nonce":2145743581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6803","nonce":2145743581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6807","nonce":2145743581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6805","nonce":2145743581}]},"public_addr":"172.21.15.134:6801/2145743581","cluster_addr":"172.21.15.134:6803/2145743581","heartbeat_back_addr":"172.21.15.134:6807/2145743581","heartbeat_front_addr":"172.21.15.134:6805/2145743581","state":["exists","up"]},{"osd":4,"uuid":"015b6da1-35ed-43f4-84c9-337d2b0e79bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6809","nonce":1867765886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6811","nonce":1867765886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6815","nonce":1867765886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6813","nonce":1867765886}]},"public_addr":"172.21.15.134:6809/1867765886","cluster_addr":"172.21.15.134:6811/1867765886","heartbeat_back_addr":"172.21.15.134:6815/1867765886","heartbeat_front_addr":"172.21.15.134:6813/1867765886","state":["exists","up"]},{"osd":5,"uuid":"86ec41d3-38df-4ee0-a320-402055f7895b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6817","nonce":3784199400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6819","nonce":3784199400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6823","nonce":3784199400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6821","nonce":3784199400}]},"public_addr":"172.21.15.134:6817/3784199400","cluster_addr":"172.21.15.134:6819/3784199400","heartbeat_back_addr":"172.21.15.134:6823/3784199400","heartbeat_front_addr":"172.21.15.134:6821/3784199400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:37:52.252795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:12.408238+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:31.280017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:47.181003+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:39:03.121900+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:39:19.383582+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/192042719":"2021-11-20T08:36:18.730761+0000","172.21.15.23:0/4146945986":"2021-11-20T08:36:18.730761+0000","172.21.15.23:6801/158654977":"2021-11-20T08:36:18.730761+0000","172.21.15.23:6800/158654977":"2021-11-20T08:36:18.730761+0000","172.21.15.23:0/992169182":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/1001097146":"2021-11-20T08:36:02.243722+0000","172.21.15.23:6800/1352671878":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/3039325239":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6801/1352671878":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/4131282839":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6800/945698991":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6801/945698991":"2021-11-20T08:35:42.544921+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-19T08:39:36.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.380+0000 7f6ace356700 1 -- 172.21.15.23:0/712166983 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6ab405c2a0 msgr2=0x7f6ab405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:36.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.380+0000 7f6ace356700 1 --2- 172.21.15.23:0/712166983 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6ab405c2a0 0x7f6ab405e760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f6ab8004e30 tx=0x7f6ab8009b30).stop 2021-11-19T08:39:36.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.380+0000 7f6ace356700 1 -- 172.21.15.23:0/712166983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ac80f5520 msgr2=0x7f6ac80fd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:36.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.380+0000 7f6ace356700 1 --2- 172.21.15.23:0/712166983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ac80f5520 0x7f6ac80fd6a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f6ac400c790 tx=0x7f6ac4007d90).stop 2021-11-19T08:39:36.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.380+0000 7f6ace356700 1 -- 172.21.15.23:0/712166983 shutdown_connections 2021-11-19T08:39:36.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.380+0000 7f6ace356700 1 --2- 172.21.15.23:0/712166983 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6ab405c2a0 0x7f6ab405e760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:36.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.380+0000 7f6ace356700 1 --2- 172.21.15.23:0/712166983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ac80f5520 0x7f6ac80fd6a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:36.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.380+0000 7f6ace356700 1 --2- 172.21.15.23:0/712166983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ac80f67c0 0x7f6ac80fbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:36.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.380+0000 7f6ace356700 1 -- 172.21.15.23:0/712166983 >> 172.21.15.23:0/712166983 conn(0x7f6ac80f0ca0 msgr2=0x7f6ac80fa210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:36.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.380+0000 7f6ace356700 1 -- 172.21.15.23:0/712166983 shutdown_connections 2021-11-19T08:39:36.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:36.380+0000 7f6ace356700 1 -- 172.21.15.23:0/712166983 wait complete. 2021-11-19T08:39:36.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:36 smithi023 conmon[33490]: cluster 2021-11-19T08:39:34.816684+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:39:36.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:36 smithi023 conmon[33490]: 154 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:36.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:36 smithi023 conmon[33490]: audit 2021-11-19T08:39:36.378823+0000 mon.smithi023 (mon.0) 438 : audit [DBG] from='client.? 172.21.15.23:0/712166983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-19T08:39:36.687 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2021-11-19T08:39:37.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:36 smithi134 conmon[36363]: cluster 2021-11-19T08:39:34.816684+0000 mgr.smithi023.ehlpcl (mgr.14164) 154 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:37.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:36 smithi134 conmon[36363]: 2021-11-19T08:39:37.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:36 smithi134 conmon[36363]: audit 2021-11-19T08:39:36.378823+0000 mon.smithi023 (mon.0) 438 : audit [DBG] from='client.? 172.21.15.23:0/712166983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-19T08:39:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.302+0000 7f8cb93a7700 1 -- 172.21.15.23:0/154489908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cb40f67e0 msgr2=0x7f8cb40f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.302+0000 7f8cb93a7700 1 --2- 172.21.15.23:0/154489908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cb40f67e0 0x7f8cb40f6c00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f8cac0099e0 tx=0x7f8cac004d10).stop 2021-11-19T08:39:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.303+0000 7f8cb93a7700 1 -- 172.21.15.23:0/154489908 shutdown_connections 2021-11-19T08:39:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.303+0000 7f8cb93a7700 1 --2- 172.21.15.23:0/154489908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cb40f7a80 0x7f8cb40f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.303+0000 7f8cb93a7700 1 --2- 172.21.15.23:0/154489908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cb40f67e0 0x7f8cb40f6c00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.303+0000 7f8cb93a7700 1 -- 172.21.15.23:0/154489908 >> 172.21.15.23:0/154489908 conn(0x7f8cb40f1f20 msgr2=0x7f8cb40f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.303+0000 7f8cb93a7700 1 -- 172.21.15.23:0/154489908 shutdown_connections 2021-11-19T08:39:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.303+0000 7f8cb93a7700 1 -- 172.21.15.23:0/154489908 wait complete. 2021-11-19T08:39:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.303+0000 7f8cb93a7700 1 Processor -- start 2021-11-19T08:39:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.303+0000 7f8cb93a7700 1 -- start start 2021-11-19T08:39:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.303+0000 7f8cb93a7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cb40f67e0 0x7f8cb406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.303+0000 7f8cb93a7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cb40f7a80 0x7f8cb4069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.303+0000 7f8cb93a7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb406a380 con 0x7f8cb40f7a80 2021-11-19T08:39:38.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.304+0000 7f8cb93a7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb406a4c0 con 0x7f8cb40f67e0 2021-11-19T08:39:38.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.304+0000 7f8cb27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cb40f7a80 0x7f8cb4069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:38.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.304+0000 7f8cb27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cb40f7a80 0x7f8cb4069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46698/0 (socket says 172.21.15.23:46698) 2021-11-19T08:39:38.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.304+0000 7f8cb27fc700 1 -- 172.21.15.23:0/282923521 learned_addr learned my addr 172.21.15.23:0/282923521 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:38.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.304+0000 7f8cb2ffd700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cb40f67e0 0x7f8cb406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:38.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.304+0000 7f8cb27fc700 1 -- 172.21.15.23:0/282923521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cb40f67e0 msgr2=0x7f8cb406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:38.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.304+0000 7f8cb27fc700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cb40f67e0 0x7f8cb406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:38.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.304+0000 7f8cb27fc700 1 -- 172.21.15.23:0/282923521 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac009640 con 0x7f8cb40f7a80 2021-11-19T08:39:38.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.304+0000 7f8cb27fc700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cb40f7a80 0x7f8cb4069e20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f8c9c00c790 tx=0x7f8c9c007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:38.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.304+0000 7f8cab7fe700 1 -- 172.21.15.23:0/282923521 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c9c002590 con 0x7f8cb40f7a80 2021-11-19T08:39:38.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.304+0000 7f8cab7fe700 1 -- 172.21.15.23:0/282923521 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c9c002ba0 con 0x7f8cb40f7a80 2021-11-19T08:39:38.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.304+0000 7f8cb93a7700 1 -- 172.21.15.23:0/282923521 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb406a7a0 con 0x7f8cb40f7a80 2021-11-19T08:39:38.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.304+0000 7f8cab7fe700 1 -- 172.21.15.23:0/282923521 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c9c013d90 con 0x7f8cb40f7a80 2021-11-19T08:39:38.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.304+0000 7f8cb93a7700 1 -- 172.21.15.23:0/282923521 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb410f220 con 0x7f8cb40f7a80 2021-11-19T08:39:38.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.305+0000 7f8cb93a7700 1 -- 172.21.15.23:0/282923521 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8cb405dcf0 con 0x7f8cb40f7a80 2021-11-19T08:39:38.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.306+0000 7f8cab7fe700 1 -- 172.21.15.23:0/282923521 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8c9c01a060 con 0x7f8cb40f7a80 2021-11-19T08:39:38.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.306+0000 7f8cab7fe700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8ca005c2a0 0x7f8ca005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:38.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.306+0000 7f8cab7fe700 1 -- 172.21.15.23:0/282923521 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8c9c075e20 con 0x7f8cb40f7a80 2021-11-19T08:39:38.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.306+0000 7f8cb2ffd700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8ca005c2a0 0x7f8ca005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:38.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.306+0000 7f8cab7fe700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305] conn(0x7f8ca0062080 0x7f8ca00644e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:38.300 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.306+0000 7f8cab7fe700 1 -- 172.21.15.23:0/282923521 --> [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8ca0064b60 con 0x7f8ca0062080 2021-11-19T08:39:38.300 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.306+0000 7f8cab7fe700 1 -- 172.21.15.23:0/282923521 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c9c078490 con 0x7f8cb40f7a80 2021-11-19T08:39:38.300 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.306+0000 7f8cb37fe700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305] conn(0x7f8ca0062080 0x7f8ca00644e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:38.300 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.306+0000 7f8cb2ffd700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8ca005c2a0 0x7f8ca005e760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8cac0099b0 tx=0x7f8cac004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:38.301 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.307+0000 7f8cb37fe700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305] conn(0x7f8ca0062080 0x7f8ca00644e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:38.301 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.307+0000 7f8cab7fe700 1 -- 172.21.15.23:0/282923521 <== osd.0 v2:172.21.15.23:6802/2382765305 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f8ca0064b60 con 0x7f8ca0062080 2021-11-19T08:39:38.307 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 -- 172.21.15.23:0/282923521 --> [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8cb40581d0 con 0x7f8ca0062080 2021-11-19T08:39:38.307 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cab7fe700 1 -- 172.21.15.23:0/282923521 <== osd.0 v2:172.21.15.23:6802/2382765305 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8cb40581d0 con 0x7f8ca0062080 2021-11-19T08:39:38.308 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 -- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305] conn(0x7f8ca0062080 msgr2=0x7f8ca00644e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:38.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305] conn(0x7f8ca0062080 0x7f8ca00644e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:38.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 -- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8ca005c2a0 msgr2=0x7f8ca005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:38.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8ca005c2a0 0x7f8ca005e760 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8cac0099b0 tx=0x7f8cac004d10).stop 2021-11-19T08:39:38.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 -- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cb40f7a80 msgr2=0x7f8cb4069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:38.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cb40f7a80 0x7f8cb4069e20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f8c9c00c790 tx=0x7f8c9c007d70).stop 2021-11-19T08:39:38.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 -- 172.21.15.23:0/282923521 shutdown_connections 2021-11-19T08:39:38.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:6802/2382765305,v1:172.21.15.23:6803/2382765305] conn(0x7f8ca0062080 0x7f8ca00644e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:38.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cb40f67e0 0x7f8cb406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:38.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8ca005c2a0 0x7f8ca005e760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 --2- 172.21.15.23:0/282923521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cb40f7a80 0x7f8cb4069e20 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 -- 172.21.15.23:0/282923521 >> 172.21.15.23:0/282923521 conn(0x7f8cb40f1f20 msgr2=0x7f8cb40fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 -- 172.21.15.23:0/282923521 shutdown_connections 2021-11-19T08:39:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:38.322+0000 7f8cb93a7700 1 -- 172.21.15.23:0/282923521 wait complete. 2021-11-19T08:39:38.641 INFO:teuthology.orchestra.run.smithi023.stdout:38654705687 2021-11-19T08:39:38.642 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2021-11-19T08:39:38.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:38 smithi023 conmon[33490]: cluster 2021-11-19T08:39:36.817024+0000 mgr.smithi023.ehlpcl (mgr.14164) 155 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:39.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:38 smithi134 conmon[36363]: cluster 2021-11-19T08:39:36.817024+0000 mgr.smithi023.ehlpcl (mgr.14164) 155 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:39.087 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:40.248 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.260+0000 7f899c05b700 1 -- 172.21.15.23:0/38330431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89940f7a80 msgr2=0x7f89940f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.260+0000 7f899c05b700 1 --2- 172.21.15.23:0/38330431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89940f7a80 0x7f89940f7ee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f8988004660 tx=0x7f8988009b30).stop 2021-11-19T08:39:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.260+0000 7f899c05b700 1 -- 172.21.15.23:0/38330431 shutdown_connections 2021-11-19T08:39:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.260+0000 7f899c05b700 1 --2- 172.21.15.23:0/38330431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89940f7a80 0x7f89940f7ee0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:40.250 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.260+0000 7f899c05b700 1 --2- 172.21.15.23:0/38330431 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89940f67e0 0x7f89940f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:40.250 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.260+0000 7f899c05b700 1 -- 172.21.15.23:0/38330431 >> 172.21.15.23:0/38330431 conn(0x7f89940f1f40 msgr2=0x7f89940f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:40.250 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.260+0000 7f899c05b700 1 -- 172.21.15.23:0/38330431 shutdown_connections 2021-11-19T08:39:40.250 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.260+0000 7f899c05b700 1 -- 172.21.15.23:0/38330431 wait complete. 2021-11-19T08:39:40.250 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.260+0000 7f899c05b700 1 Processor -- start 2021-11-19T08:39:40.251 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.260+0000 7f899c05b700 1 -- start start 2021-11-19T08:39:40.251 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f899c05b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89940f67e0 0x7f89941070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:40.251 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f899c05b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89940f7a80 0x7f89941075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:40.251 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f899c05b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8994107b60 con 0x7f89940f7a80 2021-11-19T08:39:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f899c05b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8994107ca0 con 0x7f89940f67e0 2021-11-19T08:39:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f89995f6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89940f7a80 0x7f89941075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f89995f6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89940f7a80 0x7f89941075b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46708/0 (socket says 172.21.15.23:46708) 2021-11-19T08:39:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f89995f6700 1 -- 172.21.15.23:0/1954262020 learned_addr learned my addr 172.21.15.23:0/1954262020 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f8999df7700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89940f67e0 0x7f89941070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f89995f6700 1 -- 172.21.15.23:0/1954262020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89940f67e0 msgr2=0x7f89941070c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f89995f6700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89940f67e0 0x7f89941070c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f89995f6700 1 -- 172.21.15.23:0/1954262020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8988005040 con 0x7f89940f7a80 2021-11-19T08:39:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f89995f6700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89940f7a80 0x7f89941075b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f8988009f50 tx=0x7f8988009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f89927fc700 1 -- 172.21.15.23:0/1954262020 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8988018070 con 0x7f89940f7a80 2021-11-19T08:39:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f89927fc700 1 -- 172.21.15.23:0/1954262020 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8988013410 con 0x7f89940f7a80 2021-11-19T08:39:40.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f899c05b700 1 -- 172.21.15.23:0/1954262020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89941085e0 con 0x7f89940f7a80 2021-11-19T08:39:40.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.261+0000 7f89927fc700 1 -- 172.21.15.23:0/1954262020 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f898801c3c0 con 0x7f89940f7a80 2021-11-19T08:39:40.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.262+0000 7f899c05b700 1 -- 172.21.15.23:0/1954262020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8994108b40 con 0x7f89940f7a80 2021-11-19T08:39:40.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.262+0000 7f899c05b700 1 -- 172.21.15.23:0/1954262020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f899405dd10 con 0x7f89940f7a80 2021-11-19T08:39:40.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.263+0000 7f89927fc700 1 -- 172.21.15.23:0/1954262020 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f898801d420 con 0x7f89940f7a80 2021-11-19T08:39:40.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.263+0000 7f89927fc700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f897c05c2a0 0x7f897c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:40.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.263+0000 7f89927fc700 1 -- 172.21.15.23:0/1954262020 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8988077800 con 0x7f89940f7a80 2021-11-19T08:39:40.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.263+0000 7f8999df7700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f897c05c2a0 0x7f897c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:40.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.263+0000 7f89927fc700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153] conn(0x7f897c062030 0x7f897c064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:40.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.263+0000 7f89927fc700 1 -- 172.21.15.23:0/1954262020 --> [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f897c064b10 con 0x7f897c062030 2021-11-19T08:39:40.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.263+0000 7f89927fc700 1 -- 172.21.15.23:0/1954262020 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f898807c220 con 0x7f89940f7a80 2021-11-19T08:39:40.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.263+0000 7f899a5f8700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153] conn(0x7f897c062030 0x7f897c064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:40.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.264+0000 7f8999df7700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f897c05c2a0 0x7f897c05e760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f89940f78e0 tx=0x7f898c00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:40.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.264+0000 7f899a5f8700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153] conn(0x7f897c062030 0x7f897c064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:40.258 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.264+0000 7f89927fc700 1 -- 172.21.15.23:0/1954262020 <== osd.1 v2:172.21.15.23:6810/1051412153 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f897c064b10 con 0x7f897c062030 2021-11-19T08:39:40.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 -- 172.21.15.23:0/1954262020 --> [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8994109060 con 0x7f897c062030 2021-11-19T08:39:40.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f89927fc700 1 -- 172.21.15.23:0/1954262020 <== osd.1 v2:172.21.15.23:6810/1051412153 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8994109060 con 0x7f897c062030 2021-11-19T08:39:40.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 -- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153] conn(0x7f897c062030 msgr2=0x7f897c064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:40.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153] conn(0x7f897c062030 0x7f897c064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:40.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 -- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f897c05c2a0 msgr2=0x7f897c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:40.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f897c05c2a0 0x7f897c05e760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f89940f78e0 tx=0x7f898c00b040).stop 2021-11-19T08:39:40.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 -- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89940f7a80 msgr2=0x7f89941075b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:40.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89940f7a80 0x7f89941075b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f8988009f50 tx=0x7f8988009b30).stop 2021-11-19T08:39:40.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 -- 172.21.15.23:0/1954262020 shutdown_connections 2021-11-19T08:39:40.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:6810/1051412153,v1:172.21.15.23:6811/1051412153] conn(0x7f897c062030 0x7f897c064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:40.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f89940f67e0 0x7f89941070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:40.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f897c05c2a0 0x7f897c05e760 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:40.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 --2- 172.21.15.23:0/1954262020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89940f7a80 0x7f89941075b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:40.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 -- 172.21.15.23:0/1954262020 >> 172.21.15.23:0/1954262020 conn(0x7f89940f1f40 msgr2=0x7f89940fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:40.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 -- 172.21.15.23:0/1954262020 shutdown_connections 2021-11-19T08:39:40.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:40.279+0000 7f899c05b700 1 -- 172.21.15.23:0/1954262020 wait complete. 2021-11-19T08:39:40.597 INFO:teuthology.orchestra.run.smithi023.stdout:60129542164 2021-11-19T08:39:40.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2021-11-19T08:39:40.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:40 smithi023 conmon[33490]: cluster 2021-11-19T08:39:38.817310+0000 mgr.smithi023.ehlpcl (mgr.14164) 156 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:41.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:40 smithi134 conmon[36363]: cluster 2021-11-19T08:39:38.817310+0000 mgr.smithi023.ehlpcl (mgr.14164) 156 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:41.041 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:42.207 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.219+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1578705614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbbb40607d0 msgr2=0x7fbbb40f92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:42.207 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.219+0000 7fbbb8a5c700 1 --2- 172.21.15.23:0/1578705614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbbb40607d0 0x7fbbb40f92a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fbba4004660 tx=0x7fbba4009b30).stop 2021-11-19T08:39:42.208 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.219+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1578705614 shutdown_connections 2021-11-19T08:39:42.208 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.219+0000 7fbbb8a5c700 1 --2- 172.21.15.23:0/1578705614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbb40f9790 0x7fbbb40fbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:42.208 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.219+0000 7fbbb8a5c700 1 --2- 172.21.15.23:0/1578705614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbbb40607d0 0x7fbbb40f92a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:42.208 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.219+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1578705614 >> 172.21.15.23:0/1578705614 conn(0x7fbbb40f0ca0 msgr2=0x7fbbb40f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:42.209 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.219+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1578705614 shutdown_connections 2021-11-19T08:39:42.209 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.219+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1578705614 wait complete. 2021-11-19T08:39:42.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.220+0000 7fbbb8a5c700 1 Processor -- start 2021-11-19T08:39:42.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.220+0000 7fbbb8a5c700 1 -- start start 2021-11-19T08:39:42.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.220+0000 7fbbb8a5c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbbb40607d0 0x7fbbb4106f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:42.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.220+0000 7fbbb8a5c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbb40f9790 0x7fbbb41073f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:42.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.220+0000 7fbbb8a5c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb41079a0 con 0x7fbbb40607d0 2021-11-19T08:39:42.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.220+0000 7fbbb8a5c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb4107ae0 con 0x7fbbb40f9790 2021-11-19T08:39:42.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.220+0000 7fbbb2d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbbb40607d0 0x7fbbb4106f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:42.212 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.220+0000 7fbbb2d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbbb40607d0 0x7fbbb4106f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46718/0 (socket says 172.21.15.23:46718) 2021-11-19T08:39:42.212 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.220+0000 7fbbb2d9d700 1 -- 172.21.15.23:0/1682548986 learned_addr learned my addr 172.21.15.23:0/1682548986 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:42.212 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.220+0000 7fbbb259c700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbb40f9790 0x7fbbb41073f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:42.212 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.221+0000 7fbbb2d9d700 1 -- 172.21.15.23:0/1682548986 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbb40f9790 msgr2=0x7fbbb41073f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:42.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.221+0000 7fbbb2d9d700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbb40f9790 0x7fbbb41073f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:42.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.221+0000 7fbbb2d9d700 1 -- 172.21.15.23:0/1682548986 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba4005040 con 0x7fbbb40607d0 2021-11-19T08:39:42.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.221+0000 7fbbb2d9d700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbbb40607d0 0x7fbbb4106f00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fbba4009f50 tx=0x7fbba4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:42.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.221+0000 7fbb9b7fe700 1 -- 172.21.15.23:0/1682548986 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbba4018070 con 0x7fbbb40607d0 2021-11-19T08:39:42.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.221+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1682548986 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb4108420 con 0x7fbbb40607d0 2021-11-19T08:39:42.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.221+0000 7fbb9b7fe700 1 -- 172.21.15.23:0/1682548986 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbba4013410 con 0x7fbbb40607d0 2021-11-19T08:39:42.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.221+0000 7fbb9b7fe700 1 -- 172.21.15.23:0/1682548986 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbba401c3c0 con 0x7fbbb40607d0 2021-11-19T08:39:42.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.221+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1682548986 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb4108980 con 0x7fbbb40607d0 2021-11-19T08:39:42.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.222+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1682548986 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbbb405db60 con 0x7fbbb40607d0 2021-11-19T08:39:42.217 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.222+0000 7fbb9b7fe700 1 -- 172.21.15.23:0/1682548986 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbba401d420 con 0x7fbbb40607d0 2021-11-19T08:39:42.217 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.223+0000 7fbb9b7fe700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbb9c05c2b0 0x7fbb9c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:42.218 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.223+0000 7fbb9b7fe700 1 -- 172.21.15.23:0/1682548986 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fbba4078150 con 0x7fbbb40607d0 2021-11-19T08:39:42.218 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.223+0000 7fbbb259c700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbb9c05c2b0 0x7fbb9c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:42.218 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.223+0000 7fbb9b7fe700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054] conn(0x7fbb9c062040 0x7fbb9c0644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:42.218 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.223+0000 7fbb9b7fe700 1 -- 172.21.15.23:0/1682548986 --> [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb9c064b20 con 0x7fbb9c062040 2021-11-19T08:39:42.219 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.223+0000 7fbb9b7fe700 1 -- 172.21.15.23:0/1682548986 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbba4078550 con 0x7fbbb40607d0 2021-11-19T08:39:42.219 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.223+0000 7fbbb359e700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054] conn(0x7fbb9c062040 0x7fbb9c0644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:42.219 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.223+0000 7fbbb259c700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbb9c05c2b0 0x7fbb9c05e770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fbbb40605a0 tx=0x7fbba800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:42.219 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.223+0000 7fbbb359e700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054] conn(0x7fbb9c062040 0x7fbb9c0644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:42.220 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.224+0000 7fbb9b7fe700 1 -- 172.21.15.23:0/1682548986 <== osd.2 v2:172.21.15.23:6818/954281054 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fbb9c064b20 con 0x7fbb9c062040 2021-11-19T08:39:42.223 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.238+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1682548986 --> [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbbb4108de0 con 0x7fbb9c062040 2021-11-19T08:39:42.224 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbb9b7fe700 1 -- 172.21.15.23:0/1682548986 <== osd.2 v2:172.21.15.23:6818/954281054 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbbb4108de0 con 0x7fbb9c062040 2021-11-19T08:39:42.225 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054] conn(0x7fbb9c062040 msgr2=0x7fbb9c0644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:42.225 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbbb8a5c700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054] conn(0x7fbb9c062040 0x7fbb9c0644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbb9c05c2b0 msgr2=0x7fbb9c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbbb8a5c700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbb9c05c2b0 0x7fbb9c05e770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fbbb40605a0 tx=0x7fbba800b040).stop 2021-11-19T08:39:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbbb40607d0 msgr2=0x7fbbb4106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbbb8a5c700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbbb40607d0 0x7fbbb4106f00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fbba4009f50 tx=0x7fbba4009b30).stop 2021-11-19T08:39:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1682548986 shutdown_connections 2021-11-19T08:39:42.227 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbbb8a5c700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbb9c05c2b0 0x7fbb9c05e770 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:42.227 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbbb8a5c700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbbb40607d0 0x7fbbb4106f00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:42.227 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbbb8a5c700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.23:6818/954281054,v1:172.21.15.23:6819/954281054] conn(0x7fbb9c062040 0x7fbb9c0644a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:42.227 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbbb8a5c700 1 --2- 172.21.15.23:0/1682548986 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbb40f9790 0x7fbbb41073f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:42.228 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1682548986 >> 172.21.15.23:0/1682548986 conn(0x7fbbb40f0ca0 msgr2=0x7fbbb40f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:42.228 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1682548986 shutdown_connections 2021-11-19T08:39:42.228 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:42.239+0000 7fbbb8a5c700 1 -- 172.21.15.23:0/1682548986 wait complete. 2021-11-19T08:39:42.553 INFO:teuthology.orchestra.run.smithi023.stdout:77309411345 2021-11-19T08:39:42.553 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2021-11-19T08:39:42.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:42 smithi023 conmon[33490]: cluster 2021-11-19T08:39:40.817632+0000 mgr.smithi023.ehlpcl (mgr.14164) 157 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:42.998 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:43.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:42 smithi134 conmon[36363]: cluster 2021-11-19T08:39:40.817632+0000 mgr.smithi023.ehlpcl (mgr.14164) 157 2021-11-19T08:39:43.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:42 smithi134 conmon[36363]: : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:44.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.018+0000 7f82fcc21700 1 -- 172.21.15.23:0/208095808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82f80f6980 msgr2=0x7f82f80f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:44.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.018+0000 7f82fcc21700 1 --2- 172.21.15.23:0/208095808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82f80f6980 0x7f82f80f6de0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f82ec004660 tx=0x7f82ec009b30).stop 2021-11-19T08:39:44.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.018+0000 7f82fcc21700 1 -- 172.21.15.23:0/208095808 shutdown_connections 2021-11-19T08:39:44.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.018+0000 7f82fcc21700 1 --2- 172.21.15.23:0/208095808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82f80f6980 0x7f82f80f6de0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:44.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.018+0000 7f82fcc21700 1 --2- 172.21.15.23:0/208095808 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82f80f56e0 0x7f82f80f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:44.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.018+0000 7f82fcc21700 1 -- 172.21.15.23:0/208095808 >> 172.21.15.23:0/208095808 conn(0x7f82f80f0e20 msgr2=0x7f82f80f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:44.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.018+0000 7f82fcc21700 1 -- 172.21.15.23:0/208095808 shutdown_connections 2021-11-19T08:39:44.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.018+0000 7f82fcc21700 1 -- 172.21.15.23:0/208095808 wait complete. 2021-11-19T08:39:44.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.018+0000 7f82fcc21700 1 Processor -- start 2021-11-19T08:39:44.010 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.018+0000 7f82fcc21700 1 -- start start 2021-11-19T08:39:44.010 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82fcc21700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82f80f56e0 0x7f82f80fd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:44.010 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82fcc21700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82f80f6980 0x7f82f80fbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:44.010 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82fcc21700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f80fc240 con 0x7f82f80f6980 2021-11-19T08:39:44.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82fcc21700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f80fc380 con 0x7f82f80f56e0 2021-11-19T08:39:44.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82f5d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82f80f6980 0x7f82f80fbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:44.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82f5d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82f80f6980 0x7f82f80fbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46732/0 (socket says 172.21.15.23:46732) 2021-11-19T08:39:44.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82f5d9b700 1 -- 172.21.15.23:0/2637983055 learned_addr learned my addr 172.21.15.23:0/2637983055 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:44.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82f659c700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82f80f56e0 0x7f82f80fd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:44.012 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82f5d9b700 1 -- 172.21.15.23:0/2637983055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82f80f56e0 msgr2=0x7f82f80fd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:44.012 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82f5d9b700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82f80f56e0 0x7f82f80fd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:44.012 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82f5d9b700 1 -- 172.21.15.23:0/2637983055 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec005040 con 0x7f82f80f6980 2021-11-19T08:39:44.012 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82f5d9b700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82f80f6980 0x7f82f80fbce0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f82ec009f50 tx=0x7f82ec009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:44.013 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82e6ffd700 1 -- 172.21.15.23:0/2637983055 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82ec018070 con 0x7f82f80f6980 2021-11-19T08:39:44.013 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82e6ffd700 1 -- 172.21.15.23:0/2637983055 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82ec013410 con 0x7f82f80f6980 2021-11-19T08:39:44.013 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82fcc21700 1 -- 172.21.15.23:0/2637983055 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f80fc600 con 0x7f82f80f6980 2021-11-19T08:39:44.013 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82e6ffd700 1 -- 172.21.15.23:0/2637983055 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82ec01c3c0 con 0x7f82f80f6980 2021-11-19T08:39:44.014 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.019+0000 7f82fcc21700 1 -- 172.21.15.23:0/2637983055 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f80fcb60 con 0x7f82f80f6980 2021-11-19T08:39:44.014 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.020+0000 7f82fcc21700 1 -- 172.21.15.23:0/2637983055 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f82f805dcf0 con 0x7f82f80f6980 2021-11-19T08:39:44.014 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.021+0000 7f82e6ffd700 1 -- 172.21.15.23:0/2637983055 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f82ec01d420 con 0x7f82f80f6980 2021-11-19T08:39:44.014 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.021+0000 7f82e6ffd700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f82e005c2a0 0x7f82e005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:44.015 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.021+0000 7f82e6ffd700 1 -- 172.21.15.23:0/2637983055 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f82ec0780a0 con 0x7f82f80f6980 2021-11-19T08:39:44.015 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.021+0000 7f82f659c700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f82e005c2a0 0x7f82e005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:44.015 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.022+0000 7f82e6ffd700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.134:6800/2145743581,v1:172.21.15.134:6801/2145743581] conn(0x7f82e0062030 0x7f82e0064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:44.015 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.022+0000 7f82e6ffd700 1 -- 172.21.15.23:0/2637983055 --> [v2:172.21.15.134:6800/2145743581,v1:172.21.15.134:6801/2145743581] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82e0064b10 con 0x7f82e0062030 2021-11-19T08:39:44.015 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.022+0000 7f82e6ffd700 1 -- 172.21.15.23:0/2637983055 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82ec0784a0 con 0x7f82f80f6980 2021-11-19T08:39:44.016 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.022+0000 7f82f659c700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f82e005c2a0 0x7f82e005e760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f82e8009880 tx=0x7f82e800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:44.016 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.022+0000 7f82f6d9d700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.134:6800/2145743581,v1:172.21.15.134:6801/2145743581] conn(0x7f82e0062030 0x7f82e0064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:44.016 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.022+0000 7f82f6d9d700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.134:6800/2145743581,v1:172.21.15.134:6801/2145743581] conn(0x7f82e0062030 0x7f82e0064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:44.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.023+0000 7f82e6ffd700 1 -- 172.21.15.23:0/2637983055 <== osd.3 v2:172.21.15.134:6800/2145743581 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f82e0064b10 con 0x7f82e0062030 2021-11-19T08:39:44.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.037+0000 7f82fcc21700 1 -- 172.21.15.23:0/2637983055 --> [v2:172.21.15.134:6800/2145743581,v1:172.21.15.134:6801/2145743581] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f82f8058150 con 0x7f82e0062030 2021-11-19T08:39:44.023 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82e6ffd700 1 -- 172.21.15.23:0/2637983055 <== osd.3 v2:172.21.15.134:6800/2145743581 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f82f8058150 con 0x7f82e0062030 2021-11-19T08:39:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82fcc21700 1 -- 172.21.15.23:0/2637983055 >> [v2:172.21.15.134:6800/2145743581,v1:172.21.15.134:6801/2145743581] conn(0x7f82e0062030 msgr2=0x7f82e0064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82fcc21700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.134:6800/2145743581,v1:172.21.15.134:6801/2145743581] conn(0x7f82e0062030 0x7f82e0064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82fcc21700 1 -- 172.21.15.23:0/2637983055 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f82e005c2a0 msgr2=0x7f82e005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82fcc21700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f82e005c2a0 0x7f82e005e760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f82e8009880 tx=0x7f82e800b040).stop 2021-11-19T08:39:44.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82fcc21700 1 -- 172.21.15.23:0/2637983055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82f80f6980 msgr2=0x7f82f80fbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:44.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82fcc21700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82f80f6980 0x7f82f80fbce0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f82ec009f50 tx=0x7f82ec009b30).stop 2021-11-19T08:39:44.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82fcc21700 1 -- 172.21.15.23:0/2637983055 shutdown_connections 2021-11-19T08:39:44.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82fcc21700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82f80f56e0 0x7f82f80fd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:44.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82fcc21700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.134:6800/2145743581,v1:172.21.15.134:6801/2145743581] conn(0x7f82e0062030 0x7f82e0064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:44.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82fcc21700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f82e005c2a0 0x7f82e005e760 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:44.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82fcc21700 1 --2- 172.21.15.23:0/2637983055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82f80f6980 0x7f82f80fbce0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:44.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82fcc21700 1 -- 172.21.15.23:0/2637983055 >> 172.21.15.23:0/2637983055 conn(0x7f82f80f0e20 msgr2=0x7f82f80fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:44.029 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82fcc21700 1 -- 172.21.15.23:0/2637983055 shutdown_connections 2021-11-19T08:39:44.029 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:44.038+0000 7f82fcc21700 1 -- 172.21.15.23:0/2637983055 wait complete. 2021-11-19T08:39:44.396 INFO:teuthology.orchestra.run.smithi023.stdout:111669149710 2021-11-19T08:39:44.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2021-11-19T08:39:44.636 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:44 smithi023 conmon[33490]: cluster 2021-11-19T08:39:42.817928+0000 mgr.smithi023.ehlpcl (mgr.14164) 158 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:44.844 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:45.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:44 smithi134 conmon[36363]: cluster 2021-11-19T08:39:42.817928+0000 mgr.smithi023.ehlpcl (mgr.14164) 158 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:46.097 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.110+0000 7fef4fcaa700 1 -- 172.21.15.23:0/326637304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef480f6980 msgr2=0x7fef480f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:46.098 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.110+0000 7fef4fcaa700 1 --2- 172.21.15.23:0/326637304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef480f6980 0x7fef480f6de0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fef44004660 tx=0x7fef44009b30).stop 2021-11-19T08:39:46.098 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.110+0000 7fef4fcaa700 1 -- 172.21.15.23:0/326637304 shutdown_connections 2021-11-19T08:39:46.098 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.110+0000 7fef4fcaa700 1 --2- 172.21.15.23:0/326637304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef480f6980 0x7fef480f6de0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:46.099 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.110+0000 7fef4fcaa700 1 --2- 172.21.15.23:0/326637304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef480f56e0 0x7fef480f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-19T08:39:46.099 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.110+0000 7fef4fcaa700 1 -- 172.21.15.23:0/326637304 >> 172.21.15.23:0/326637304 conn(0x7fef480f0e20 msgr2=0x7fef480f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:46.099 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.110+0000 7fef4fcaa700 1 -- 172.21.15.23:0/326637304 shutdown_connections 2021-11-19T08:39:46.099 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.110+0000 7fef4fcaa700 1 -- 172.21.15.23:0/326637304 wait complete. 2021-11-19T08:39:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.111+0000 7fef4fcaa700 1 Processor -- start 2021-11-19T08:39:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.111+0000 7fef4fcaa700 1 -- start start 2021-11-19T08:39:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.111+0000 7fef4fcaa700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef480f56e0 0x7fef48107100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.111+0000 7fef4fcaa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef480f6980 0x7fef481075f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.111+0000 7fef4fcaa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef48107c30 con 0x7fef480f6980 2021-11-19T08:39:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.111+0000 7fef4fcaa700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef481084e0 con 0x7fef480f56e0 2021-11-19T08:39:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.111+0000 7fef4d245700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef480f6980 0x7fef481075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.111+0000 7fef4d245700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef480f6980 0x7fef481075f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46744/0 (socket says 172.21.15.23:46744) 2021-11-19T08:39:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.111+0000 7fef4d245700 1 -- 172.21.15.23:0/1709837397 learned_addr learned my addr 172.21.15.23:0/1709837397 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:46.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.112+0000 7fef4da46700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef480f56e0 0x7fef48107100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:46.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.112+0000 7fef4d245700 1 -- 172.21.15.23:0/1709837397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef480f56e0 msgr2=0x7fef48107100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:46.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.112+0000 7fef4d245700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef480f56e0 0x7fef48107100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:46.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.112+0000 7fef4d245700 1 -- 172.21.15.23:0/1709837397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef44005040 con 0x7fef480f6980 2021-11-19T08:39:46.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.112+0000 7fef4da46700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef480f56e0 0x7fef48107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:39:46.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.112+0000 7fef4d245700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef480f6980 0x7fef481075f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fef44009830 tx=0x7fef44009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:46.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.112+0000 7fef3e7fc700 1 -- 172.21.15.23:0/1709837397 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef44018070 con 0x7fef480f6980 2021-11-19T08:39:46.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.112+0000 7fef3e7fc700 1 -- 172.21.15.23:0/1709837397 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef44013410 con 0x7fef480f6980 2021-11-19T08:39:46.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.112+0000 7fef4fcaa700 1 -- 172.21.15.23:0/1709837397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef48108760 con 0x7fef480f6980 2021-11-19T08:39:46.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.112+0000 7fef3e7fc700 1 -- 172.21.15.23:0/1709837397 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef4401c620 con 0x7fef480f6980 2021-11-19T08:39:46.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.112+0000 7fef4fcaa700 1 -- 172.21.15.23:0/1709837397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef48108cc0 con 0x7fef480f6980 2021-11-19T08:39:46.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.113+0000 7fef4fcaa700 1 -- 172.21.15.23:0/1709837397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fef4805dcf0 con 0x7fef480f6980 2021-11-19T08:39:46.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.113+0000 7fef3e7fc700 1 -- 172.21.15.23:0/1709837397 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fef4401d420 con 0x7fef480f6980 2021-11-19T08:39:46.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.114+0000 7fef3e7fc700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fef3405c2a0 0x7fef3405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:46.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.114+0000 7fef3e7fc700 1 -- 172.21.15.23:0/1709837397 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fef44077a70 con 0x7fef480f6980 2021-11-19T08:39:46.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.114+0000 7fef4da46700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fef3405c2a0 0x7fef3405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:46.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.114+0000 7fef3e7fc700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.134:6808/1867765886,v1:172.21.15.134:6809/1867765886] conn(0x7fef34062030 0x7fef34064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:46.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.114+0000 7fef3e7fc700 1 -- 172.21.15.23:0/1709837397 --> [v2:172.21.15.134:6808/1867765886,v1:172.21.15.134:6809/1867765886] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef34064b10 con 0x7fef34062030 2021-11-19T08:39:46.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.114+0000 7fef3e7fc700 1 -- 172.21.15.23:0/1709837397 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef4407c220 con 0x7fef480f6980 2021-11-19T08:39:46.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.114+0000 7fef4e247700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.134:6808/1867765886,v1:172.21.15.134:6809/1867765886] conn(0x7fef34062030 0x7fef34064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:46.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.114+0000 7fef4da46700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fef3405c2a0 0x7fef3405e760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fef480f67e0 tx=0x7fef38008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:46.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.115+0000 7fef4e247700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.134:6808/1867765886,v1:172.21.15.134:6809/1867765886] conn(0x7fef34062030 0x7fef34064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:46.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.115+0000 7fef3e7fc700 1 -- 172.21.15.23:0/1709837397 <== osd.4 v2:172.21.15.134:6808/1867765886 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fef34064b10 con 0x7fef34062030 2021-11-19T08:39:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.131+0000 7fef4fcaa700 1 -- 172.21.15.23:0/1709837397 --> [v2:172.21.15.134:6808/1867765886,v1:172.21.15.134:6809/1867765886] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fef481091e0 con 0x7fef34062030 2021-11-19T08:39:46.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.132+0000 7fef3e7fc700 1 -- 172.21.15.23:0/1709837397 <== osd.4 v2:172.21.15.134:6808/1867765886 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fef481091e0 con 0x7fef34062030 2021-11-19T08:39:46.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.132+0000 7fef4fcaa700 1 -- 172.21.15.23:0/1709837397 >> [v2:172.21.15.134:6808/1867765886,v1:172.21.15.134:6809/1867765886] conn(0x7fef34062030 msgr2=0x7fef34064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:46.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.132+0000 7fef4fcaa700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.134:6808/1867765886,v1:172.21.15.134:6809/1867765886] conn(0x7fef34062030 0x7fef34064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:46.120 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.132+0000 7fef4fcaa700 1 -- 172.21.15.23:0/1709837397 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fef3405c2a0 msgr2=0x7fef3405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:46.120 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.132+0000 7fef4fcaa700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fef3405c2a0 0x7fef3405e760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fef480f67e0 tx=0x7fef38008040).stop 2021-11-19T08:39:46.120 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.132+0000 7fef4fcaa700 1 -- 172.21.15.23:0/1709837397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef480f6980 msgr2=0x7fef481075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:46.120 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.132+0000 7fef4fcaa700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef480f6980 0x7fef481075f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fef44009830 tx=0x7fef44009b30).stop 2021-11-19T08:39:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.132+0000 7fef4fcaa700 1 -- 172.21.15.23:0/1709837397 shutdown_connections 2021-11-19T08:39:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.132+0000 7fef4fcaa700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.134:6808/1867765886,v1:172.21.15.134:6809/1867765886] conn(0x7fef34062030 0x7fef34064490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.132+0000 7fef4fcaa700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef480f56e0 0x7fef48107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.132+0000 7fef4fcaa700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fef3405c2a0 0x7fef3405e760 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0x7fef480f67e0 tx=0x7fef38008040).stop 2021-11-19T08:39:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.132+0000 7fef4fcaa700 1 --2- 172.21.15.23:0/1709837397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef480f6980 0x7fef481075f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.132+0000 7fef4fcaa700 1 -- 172.21.15.23:0/1709837397 >> 172.21.15.23:0/1709837397 conn(0x7fef480f0e20 msgr2=0x7fef480603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.132+0000 7fef4fcaa700 1 -- 172.21.15.23:0/1709837397 shutdown_connections 2021-11-19T08:39:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:46.133+0000 7fef4fcaa700 1 -- 172.21.15.23:0/1709837397 wait complete. 2021-11-19T08:39:46.444 INFO:teuthology.orchestra.run.smithi023.stdout:133143986187 2021-11-19T08:39:46.444 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2021-11-19T08:39:46.889 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:46.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:46 smithi023 conmon[33490]: cluster 2021-11-19T08:39:44.818201 2021-11-19T08:39:46.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:46 smithi023 conmon[33490]: +0000 mgr.smithi023.ehlpcl (mgr.14164) 159 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:47.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:46 smithi134 conmon[36363]: cluster 2021-11-19T08:39:44.818201+0000 mgr.smithi023.ehlpcl (mgr.14164) 159 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:47.987 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.000+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3733354311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0780f4aa0 msgr2=0x7fc0780f4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:47.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.000+0000 7fc07d2a0700 1 --2- 172.21.15.23:0/3733354311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0780f4aa0 0x7fc0780f4ec0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fc068004660 tx=0x7fc068009b30).stop 2021-11-19T08:39:47.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.000+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3733354311 shutdown_connections 2021-11-19T08:39:47.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.000+0000 7fc07d2a0700 1 --2- 172.21.15.23:0/3733354311 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0780f5440 0x7fc0780f7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:47.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.000+0000 7fc07d2a0700 1 --2- 172.21.15.23:0/3733354311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0780f4aa0 0x7fc0780f4ec0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:47.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.000+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3733354311 >> 172.21.15.23:0/3733354311 conn(0x7fc0780f05e0 msgr2=0x7fc0780f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:47.989 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.000+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3733354311 shutdown_connections 2021-11-19T08:39:47.989 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.000+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3733354311 wait complete. 2021-11-19T08:39:47.989 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.001+0000 7fc07d2a0700 1 Processor -- start 2021-11-19T08:39:47.989 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.001+0000 7fc07d2a0700 1 -- start start 2021-11-19T08:39:47.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.001+0000 7fc07d2a0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0780f4aa0 0x7fc078107100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:47.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.001+0000 7fc07d2a0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0780f5440 0x7fc0781075f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:47.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.001+0000 7fc07d2a0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc078107ba0 con 0x7fc0780f4aa0 2021-11-19T08:39:47.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.001+0000 7fc07d2a0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc078107ce0 con 0x7fc0780f5440 2021-11-19T08:39:47.991 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.001+0000 7fc076ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0780f4aa0 0x7fc078107100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:47.991 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.001+0000 7fc076ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0780f4aa0 0x7fc078107100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46754/0 (socket says 172.21.15.23:46754) 2021-11-19T08:39:47.991 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.001+0000 7fc076ffd700 1 -- 172.21.15.23:0/3789353695 learned_addr learned my addr 172.21.15.23:0/3789353695 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:47.992 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.002+0000 7fc0767fc700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0780f5440 0x7fc0781075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:47.992 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.002+0000 7fc076ffd700 1 -- 172.21.15.23:0/3789353695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0780f5440 msgr2=0x7fc0781075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:47.992 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.002+0000 7fc076ffd700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0780f5440 0x7fc0781075f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:47.992 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.002+0000 7fc076ffd700 1 -- 172.21.15.23:0/3789353695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc068005040 con 0x7fc0780f4aa0 2021-11-19T08:39:47.993 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.002+0000 7fc076ffd700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0780f4aa0 0x7fc078107100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fc068004e30 tx=0x7fc068009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:47.993 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.002+0000 7fc06f7fe700 1 -- 172.21.15.23:0/3789353695 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc068018070 con 0x7fc0780f4aa0 2021-11-19T08:39:47.993 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.002+0000 7fc06f7fe700 1 -- 172.21.15.23:0/3789353695 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc068013410 con 0x7fc0780f4aa0 2021-11-19T08:39:47.993 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.002+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3789353695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc078108620 con 0x7fc0780f4aa0 2021-11-19T08:39:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.002+0000 7fc06f7fe700 1 -- 172.21.15.23:0/3789353695 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc06801c3c0 con 0x7fc0780f4aa0 2021-11-19T08:39:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.002+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3789353695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc078108b80 con 0x7fc0780f4aa0 2021-11-19T08:39:47.995 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.003+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3789353695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc07805dcf0 con 0x7fc0780f4aa0 2021-11-19T08:39:47.996 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.003+0000 7fc06f7fe700 1 -- 172.21.15.23:0/3789353695 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc06801d420 con 0x7fc0780f4aa0 2021-11-19T08:39:47.996 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.004+0000 7fc06f7fe700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc06405c2a0 0x7fc06405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:47.996 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.004+0000 7fc06f7fe700 1 -- 172.21.15.23:0/3789353695 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc068077820 con 0x7fc0780f4aa0 2021-11-19T08:39:47.996 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.004+0000 7fc0767fc700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc06405c2a0 0x7fc06405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:47.997 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.004+0000 7fc06f7fe700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.134:6816/3784199400,v1:172.21.15.134:6817/3784199400] conn(0x7fc064062030 0x7fc064064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:47.997 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.004+0000 7fc06f7fe700 1 -- 172.21.15.23:0/3789353695 --> [v2:172.21.15.134:6816/3784199400,v1:172.21.15.134:6817/3784199400] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc064064b10 con 0x7fc064062030 2021-11-19T08:39:47.997 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.004+0000 7fc06f7fe700 1 -- 172.21.15.23:0/3789353695 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc06807c220 con 0x7fc0780f4aa0 2021-11-19T08:39:47.997 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.004+0000 7fc0777fe700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.134:6816/3784199400,v1:172.21.15.134:6817/3784199400] conn(0x7fc064062030 0x7fc064064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:47.998 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.004+0000 7fc0767fc700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc06405c2a0 0x7fc06405e760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fc0780f5880 tx=0x7fc060009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:47.998 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.004+0000 7fc0777fe700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.134:6816/3784199400,v1:172.21.15.134:6817/3784199400] conn(0x7fc064062030 0x7fc064064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:47.998 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.005+0000 7fc06f7fe700 1 -- 172.21.15.23:0/3789353695 <== osd.5 v2:172.21.15.134:6816/3784199400 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fc064064b10 con 0x7fc064062030 2021-11-19T08:39:48.004 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.019+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3789353695 --> [v2:172.21.15.134:6816/3784199400,v1:172.21.15.134:6817/3784199400] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc0781090a0 con 0x7fc064062030 2021-11-19T08:39:48.005 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc06f7fe700 1 -- 172.21.15.23:0/3789353695 <== osd.5 v2:172.21.15.134:6816/3784199400 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc0781090a0 con 0x7fc064062030 2021-11-19T08:39:48.006 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3789353695 >> [v2:172.21.15.134:6816/3784199400,v1:172.21.15.134:6817/3784199400] conn(0x7fc064062030 msgr2=0x7fc064064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:48.006 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc07d2a0700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.134:6816/3784199400,v1:172.21.15.134:6817/3784199400] conn(0x7fc064062030 0x7fc064064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:48.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3789353695 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc06405c2a0 msgr2=0x7fc06405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:48.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc07d2a0700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc06405c2a0 0x7fc06405e760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fc0780f5880 tx=0x7fc060009be0).stop 2021-11-19T08:39:48.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3789353695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0780f4aa0 msgr2=0x7fc078107100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:48.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc07d2a0700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0780f4aa0 0x7fc078107100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fc068004e30 tx=0x7fc068009b30).stop 2021-11-19T08:39:48.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3789353695 shutdown_connections 2021-11-19T08:39:48.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc07d2a0700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.134:6816/3784199400,v1:172.21.15.134:6817/3784199400] conn(0x7fc064062030 0x7fc064064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:48.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc07d2a0700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc06405c2a0 0x7fc06405e760 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:48.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc07d2a0700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0780f4aa0 0x7fc078107100 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:48.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc07d2a0700 1 --2- 172.21.15.23:0/3789353695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0780f5440 0x7fc0781075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:48.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3789353695 >> 172.21.15.23:0/3789353695 conn(0x7fc0780f05e0 msgr2=0x7fc0780fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:48.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3789353695 shutdown_connections 2021-11-19T08:39:48.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:48.020+0000 7fc07d2a0700 1 -- 172.21.15.23:0/3789353695 wait complete. 2021-11-19T08:39:48.390 INFO:teuthology.orchestra.run.smithi023.stdout:150323855369 2021-11-19T08:39:48.390 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-11-19T08:39:48.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:48 smithi023 conmon[33490]: cluster 2021-11-19T08:39:46.818504+0000 mgr.smithi023.ehlpcl (mgr.14164) 160 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:48.833 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:49.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:48 smithi134 conmon[36363]: cluster 2021-11-19T08:39:46.818504+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:39:49.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:48 smithi134 conmon[36363]: 160 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:49.866 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.879+0000 7f4b2d326700 1 -- 172.21.15.23:0/303925251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b280f7a80 msgr2=0x7f4b280f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:49.867 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.879+0000 7f4b2d326700 1 --2- 172.21.15.23:0/303925251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b280f7a80 0x7f4b280f7ee0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f4b18004660 tx=0x7f4b18009b30).stop 2021-11-19T08:39:49.867 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.879+0000 7f4b2d326700 1 -- 172.21.15.23:0/303925251 shutdown_connections 2021-11-19T08:39:49.867 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.879+0000 7f4b2d326700 1 --2- 172.21.15.23:0/303925251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b280f7a80 0x7f4b280f7ee0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:49.867 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.879+0000 7f4b2d326700 1 --2- 172.21.15.23:0/303925251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b280f67e0 0x7f4b280f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-19T08:39:49.868 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.879+0000 7f4b2d326700 1 -- 172.21.15.23:0/303925251 >> 172.21.15.23:0/303925251 conn(0x7f4b280f1f20 msgr2=0x7f4b280f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:49.868 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.879+0000 7f4b2d326700 1 -- 172.21.15.23:0/303925251 shutdown_connections 2021-11-19T08:39:49.868 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.879+0000 7f4b2d326700 1 -- 172.21.15.23:0/303925251 wait complete. 2021-11-19T08:39:49.869 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.880+0000 7f4b2d326700 1 Processor -- start 2021-11-19T08:39:49.869 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.880+0000 7f4b2d326700 1 -- start start 2021-11-19T08:39:49.869 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.880+0000 7f4b2d326700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b280f67e0 0x7f4b281070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:49.869 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.880+0000 7f4b2d326700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b280f7a80 0x7f4b281075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:49.869 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.880+0000 7f4b2d326700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b28107c00 con 0x7f4b280f67e0 2021-11-19T08:39:49.870 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.880+0000 7f4b2d326700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b281084b0 con 0x7f4b280f7a80 2021-11-19T08:39:49.870 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.880+0000 7f4b26ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b280f67e0 0x7f4b281070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:49.870 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.880+0000 7f4b26ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b280f67e0 0x7f4b281070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46766/0 (socket says 172.21.15.23:46766) 2021-11-19T08:39:49.870 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.880+0000 7f4b26ffd700 1 -- 172.21.15.23:0/868900762 learned_addr learned my addr 172.21.15.23:0/868900762 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:49.871 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.881+0000 7f4b267fc700 1 --2- 172.21.15.23:0/868900762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b280f7a80 0x7f4b281075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:49.871 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.881+0000 7f4b26ffd700 1 -- 172.21.15.23:0/868900762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b280f7a80 msgr2=0x7f4b281075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:49.871 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.881+0000 7f4b26ffd700 1 --2- 172.21.15.23:0/868900762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b280f7a80 0x7f4b281075c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:49.872 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.881+0000 7f4b26ffd700 1 -- 172.21.15.23:0/868900762 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b18005040 con 0x7f4b280f67e0 2021-11-19T08:39:49.872 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.881+0000 7f4b26ffd700 1 --2- 172.21.15.23:0/868900762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b280f67e0 0x7f4b281070d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4b100093f0 tx=0x7f4b10006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:49.872 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.881+0000 7f4b1f7fe700 1 -- 172.21.15.23:0/868900762 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b10009ea0 con 0x7f4b280f67e0 2021-11-19T08:39:49.872 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.881+0000 7f4b1f7fe700 1 -- 172.21.15.23:0/868900762 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b10013af0 con 0x7f4b280f67e0 2021-11-19T08:39:49.872 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.881+0000 7f4b1f7fe700 1 -- 172.21.15.23:0/868900762 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b1000dad0 con 0x7f4b280f67e0 2021-11-19T08:39:49.873 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.881+0000 7f4b2d326700 1 -- 172.21.15.23:0/868900762 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b28108790 con 0x7f4b280f67e0 2021-11-19T08:39:49.874 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.881+0000 7f4b2d326700 1 -- 172.21.15.23:0/868900762 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b28108d50 con 0x7f4b280f67e0 2021-11-19T08:39:49.875 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.882+0000 7f4b2d326700 1 -- 172.21.15.23:0/868900762 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b28059100 con 0x7f4b280f67e0 2021-11-19T08:39:49.875 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.882+0000 7f4b1f7fe700 1 -- 172.21.15.23:0/868900762 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4b1001a060 con 0x7f4b280f67e0 2021-11-19T08:39:49.875 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.883+0000 7f4b1f7fe700 1 --2- 172.21.15.23:0/868900762 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4b1405c2a0 0x7f4b1405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:49.875 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.883+0000 7f4b1f7fe700 1 -- 172.21.15.23:0/868900762 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4b10075bb0 con 0x7f4b280f67e0 2021-11-19T08:39:49.876 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.883+0000 7f4b267fc700 1 --2- 172.21.15.23:0/868900762 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4b1405c2a0 0x7f4b1405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:49.876 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.884+0000 7f4b267fc700 1 --2- 172.21.15.23:0/868900762 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4b1405c2a0 0x7f4b1405e760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f4b18013580 tx=0x7f4b18009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:49.876 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:49.887+0000 7f4b1f7fe700 1 -- 172.21.15.23:0/868900762 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b1007c080 con 0x7f4b280f67e0 2021-11-19T08:39:50.030 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:50.044+0000 7f4b2d326700 1 -- 172.21.15.23:0/868900762 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4b280ff010 con 0x7f4b280f67e0 2021-11-19T08:39:50.030 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:50.045+0000 7f4b1f7fe700 1 -- 172.21.15.23:0/868900762 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4b10048bb0 con 0x7f4b280f67e0 2021-11-19T08:39:50.030 INFO:teuthology.orchestra.run.smithi023.stdout:38654705689 2021-11-19T08:39:50.031 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:50.046+0000 7f4b2d326700 1 -- 172.21.15.23:0/868900762 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4b1405c2a0 msgr2=0x7f4b1405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:50.032 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:50.046+0000 7f4b2d326700 1 --2- 172.21.15.23:0/868900762 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4b1405c2a0 0x7f4b1405e760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f4b18013580 tx=0x7f4b18009b30).stop 2021-11-19T08:39:50.032 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:50.046+0000 7f4b2d326700 1 -- 172.21.15.23:0/868900762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b280f67e0 msgr2=0x7f4b281070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:50.032 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:50.046+0000 7f4b2d326700 1 --2- 172.21.15.23:0/868900762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b280f67e0 0x7f4b281070d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4b100093f0 tx=0x7f4b10006ec0).stop 2021-11-19T08:39:50.032 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:50.046+0000 7f4b2d326700 1 -- 172.21.15.23:0/868900762 shutdown_connections 2021-11-19T08:39:50.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:50.046+0000 7f4b2d326700 1 --2- 172.21.15.23:0/868900762 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4b1405c2a0 0x7f4b1405e760 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:50.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:50.046+0000 7f4b2d326700 1 --2- 172.21.15.23:0/868900762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b280f67e0 0x7f4b281070d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:50.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:50.046+0000 7f4b2d326700 1 --2- 172.21.15.23:0/868900762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b280f7a80 0x7f4b281075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:50.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:50.046+0000 7f4b2d326700 1 -- 172.21.15.23:0/868900762 >> 172.21.15.23:0/868900762 conn(0x7f4b280f1f20 msgr2=0x7f4b280fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:50.034 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:50.046+0000 7f4b2d326700 1 -- 172.21.15.23:0/868900762 shutdown_connections 2021-11-19T08:39:50.034 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:50.046+0000 7f4b2d326700 1 -- 172.21.15.23:0/868900762 wait complete. 2021-11-19T08:39:50.431 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705687 got 38654705689 for osd.0 2021-11-19T08:39:50.432 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.1 2021-11-19T08:39:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:50.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:50 smithi023 conmon[33490]: cluster 2021-11-19T08:39:48.818835+0000 2021-11-19T08:39:50.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:50 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 161 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:50.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:50 smithi023 conmon[33490]: audit 2021-11-19T08:39:50.045897+0000 mon.smithi023 (mon.0) 439 : audit [DBG] from='client.? 172.21.15.23:0/868900762' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-11-19T08:39:51.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:50 smithi134 conmon[36363]: cluster 2021-11-19T08:39:48.818835+0000 mgr.smithi023.ehlpcl (mgr.14164) 161 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:51.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:50 smithi134 conmon[36363]: audit 2021-11-19T08 2021-11-19T08:39:51.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:50 smithi134 conmon[36363]: :39:50.045897+0000 mon.smithi023 (mon.0) 439 : audit [DBG] from='client.? 172.21.15.23:0/868900762' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-11-19T08:39:52.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.113+0000 7f54a7889700 1 -- 172.21.15.23:0/4206164923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a0060810 msgr2=0x7f54a00f9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:52.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.113+0000 7f54a7889700 1 --2- 172.21.15.23:0/4206164923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a0060810 0x7f54a00f9420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f5498004660 tx=0x7f5498009b30).stop 2021-11-19T08:39:52.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.113+0000 7f54a7889700 1 -- 172.21.15.23:0/4206164923 shutdown_connections 2021-11-19T08:39:52.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.113+0000 7f54a7889700 1 --2- 172.21.15.23:0/4206164923 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54a00f99a0 0x7f54a00fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:52.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.113+0000 7f54a7889700 1 --2- 172.21.15.23:0/4206164923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a0060810 0x7f54a00f9420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:52.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.113+0000 7f54a7889700 1 -- 172.21.15.23:0/4206164923 >> 172.21.15.23:0/4206164923 conn(0x7f54a00f0e20 msgr2=0x7f54a00f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:52.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.113+0000 7f54a7889700 1 -- 172.21.15.23:0/4206164923 shutdown_connections 2021-11-19T08:39:52.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.113+0000 7f54a7889700 1 -- 172.21.15.23:0/4206164923 wait complete. 2021-11-19T08:39:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.113+0000 7f54a7889700 1 Processor -- start 2021-11-19T08:39:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.113+0000 7f54a7889700 1 -- start start 2021-11-19T08:39:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f54a7889700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a0060810 0x7f54a00f6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f54a7889700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54a00f99a0 0x7f54a00f5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f54a7889700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a00f5880 con 0x7f54a0060810 2021-11-19T08:39:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f54a7889700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a00f59c0 con 0x7f54a00f99a0 2021-11-19T08:39:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f54a5625700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a0060810 0x7f54a00f6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f54a5625700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a0060810 0x7f54a00f6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46776/0 (socket says 172.21.15.23:46776) 2021-11-19T08:39:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f54a5625700 1 -- 172.21.15.23:0/3397418269 learned_addr learned my addr 172.21.15.23:0/3397418269 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f54a4e24700 1 --2- 172.21.15.23:0/3397418269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54a00f99a0 0x7f54a00f5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f54a5625700 1 -- 172.21.15.23:0/3397418269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54a00f99a0 msgr2=0x7f54a00f5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:52.110 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f54a5625700 1 --2- 172.21.15.23:0/3397418269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54a00f99a0 0x7f54a00f5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:52.110 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f54a5625700 1 -- 172.21.15.23:0/3397418269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5498005040 con 0x7f54a0060810 2021-11-19T08:39:52.110 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f54a5625700 1 --2- 172.21.15.23:0/3397418269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a0060810 0x7f54a00f6ea0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f5498009f50 tx=0x7f5498009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:52.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f549dffb700 1 -- 172.21.15.23:0/3397418269 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5498018070 con 0x7f54a0060810 2021-11-19T08:39:52.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f549dffb700 1 -- 172.21.15.23:0/3397418269 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5498013410 con 0x7f54a0060810 2021-11-19T08:39:52.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f54a7889700 1 -- 172.21.15.23:0/3397418269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a00f5c40 con 0x7f54a0060810 2021-11-19T08:39:52.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.114+0000 7f549dffb700 1 -- 172.21.15.23:0/3397418269 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f549801c3c0 con 0x7f54a0060810 2021-11-19T08:39:52.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.115+0000 7f54a7889700 1 -- 172.21.15.23:0/3397418269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a00f61a0 con 0x7f54a0060810 2021-11-19T08:39:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.115+0000 7f54a7889700 1 -- 172.21.15.23:0/3397418269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a0045ad0 con 0x7f54a0060810 2021-11-19T08:39:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.116+0000 7f549dffb700 1 -- 172.21.15.23:0/3397418269 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f549801d420 con 0x7f54a0060810 2021-11-19T08:39:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.116+0000 7f549dffb700 1 --2- 172.21.15.23:0/3397418269 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f548805c2a0 0x7f548805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.116+0000 7f549dffb700 1 -- 172.21.15.23:0/3397418269 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f5498077810 con 0x7f54a0060810 2021-11-19T08:39:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.117+0000 7f54a4e24700 1 --2- 172.21.15.23:0/3397418269 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f548805c2a0 0x7f548805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.117+0000 7f54a4e24700 1 --2- 172.21.15.23:0/3397418269 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f548805c2a0 0x7f548805e760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f54a00605e0 tx=0x7f549400b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:52.114 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.120+0000 7f549dffb700 1 -- 172.21.15.23:0/3397418269 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f549804a7d0 con 0x7f54a0060810 2021-11-19T08:39:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.280+0000 7f54a7889700 1 -- 172.21.15.23:0/3397418269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f54a0027a00 con 0x7f54a0060810 2021-11-19T08:39:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.280+0000 7f549dffb700 1 -- 172.21.15.23:0/3397418269 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f549804ddf0 con 0x7f54a0060810 2021-11-19T08:39:52.266 INFO:teuthology.orchestra.run.smithi023.stdout:60129542166 2021-11-19T08:39:52.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.282+0000 7f54a7889700 1 -- 172.21.15.23:0/3397418269 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f548805c2a0 msgr2=0x7f548805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:52.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.282+0000 7f54a7889700 1 --2- 172.21.15.23:0/3397418269 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f548805c2a0 0x7f548805e760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f54a00605e0 tx=0x7f549400b040).stop 2021-11-19T08:39:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.282+0000 7f54a7889700 1 -- 172.21.15.23:0/3397418269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a0060810 msgr2=0x7f54a00f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.282+0000 7f54a7889700 1 --2- 172.21.15.23:0/3397418269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a0060810 0x7f54a00f6ea0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f5498009f50 tx=0x7f5498009b30).stop 2021-11-19T08:39:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.282+0000 7f54a7889700 1 -- 172.21.15.23:0/3397418269 shutdown_connections 2021-11-19T08:39:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.282+0000 7f54a7889700 1 --2- 172.21.15.23:0/3397418269 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f548805c2a0 0x7f548805e760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.282+0000 7f54a7889700 1 --2- 172.21.15.23:0/3397418269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a0060810 0x7f54a00f6ea0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:52.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.282+0000 7f54a7889700 1 --2- 172.21.15.23:0/3397418269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54a00f99a0 0x7f54a00f5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:52.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.282+0000 7f54a7889700 1 -- 172.21.15.23:0/3397418269 >> 172.21.15.23:0/3397418269 conn(0x7f54a00f0e20 msgr2=0x7f54a00f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:52.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.282+0000 7f54a7889700 1 -- 172.21.15.23:0/3397418269 shutdown_connections 2021-11-19T08:39:52.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:52.282+0000 7f54a7889700 1 -- 172.21.15.23:0/3397418269 wait complete. 2021-11-19T08:39:52.577 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542164 got 60129542166 for osd.1 2021-11-19T08:39:52.577 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.2 2021-11-19T08:39:52.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:52 smithi023 conmon[33490]: cluster 2021-11-19T08:39:50.819132+0000 mgr.smithi023.ehlpcl (mgr.14164) 162 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:52.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:52 smithi023 conmon[33490]: audit 2021-11-19T08:39:52.281264+0000 mon.smithi023 (mon.0) 440 : audit [DBG] from='client.? 172.21.15.23:0/3397418269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-11-19T08:39:53.021 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:53.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:52 smithi134 conmon[36363]: cluster 2021-11-19T08:39:50.819132+0000 mgr.smithi023.ehlpcl (mgr.14164) 162 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:53.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:52 smithi134 conmon[36363]: 2021-11-19T08:39:53.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:52 smithi134 conmon[36363]: audit 2021-11-19T08:39:52.281264+0000 mon.smithi023 (mon.0) 440 : audit [DBG] from='client.? 172.21.15.23:0/3397418269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-11-19T08:39:54.171 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.184+0000 7fd801cc9700 1 -- 172.21.15.23:0/1978928299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7fc060810 msgr2=0x7fd7fc0f9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:54.171 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.184+0000 7fd801cc9700 1 --2- 172.21.15.23:0/1978928299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7fc060810 0x7fd7fc0f9420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd7ec004660 tx=0x7fd7ec009b30).stop 2021-11-19T08:39:54.172 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.184+0000 7fd801cc9700 1 -- 172.21.15.23:0/1978928299 shutdown_connections 2021-11-19T08:39:54.172 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.184+0000 7fd801cc9700 1 --2- 172.21.15.23:0/1978928299 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7fc0f99a0 0x7fd7fc0fbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:54.172 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.184+0000 7fd801cc9700 1 --2- 172.21.15.23:0/1978928299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7fc060810 0x7fd7fc0f9420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:54.172 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.184+0000 7fd801cc9700 1 -- 172.21.15.23:0/1978928299 >> 172.21.15.23:0/1978928299 conn(0x7fd7fc0f0e20 msgr2=0x7fd7fc0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:54.173 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.184+0000 7fd801cc9700 1 -- 172.21.15.23:0/1978928299 shutdown_connections 2021-11-19T08:39:54.173 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.184+0000 7fd801cc9700 1 -- 172.21.15.23:0/1978928299 wait complete. 2021-11-19T08:39:54.173 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd801cc9700 1 Processor -- start 2021-11-19T08:39:54.173 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd801cc9700 1 -- start start 2021-11-19T08:39:54.174 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd801cc9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7fc060810 0x7fd7fc104e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:54.174 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd801cc9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7fc0f99a0 0x7fd7fc105370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:54.174 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd801cc9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc105970 con 0x7fd7fc060810 2021-11-19T08:39:54.174 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd801cc9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc105ab0 con 0x7fd7fc0f99a0 2021-11-19T08:39:54.175 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd7fb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7fc060810 0x7fd7fc104e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:54.175 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd7fb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7fc060810 0x7fd7fc104e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46786/0 (socket says 172.21.15.23:46786) 2021-11-19T08:39:54.175 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd7fb7fe700 1 -- 172.21.15.23:0/881994789 learned_addr learned my addr 172.21.15.23:0/881994789 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:54.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd7f3fff700 1 --2- 172.21.15.23:0/881994789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7fc0f99a0 0x7fd7fc105370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:54.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd7fb7fe700 1 -- 172.21.15.23:0/881994789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7fc0f99a0 msgr2=0x7fd7fc105370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:54.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd7fb7fe700 1 --2- 172.21.15.23:0/881994789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7fc0f99a0 0x7fd7fc105370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:54.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd7fb7fe700 1 -- 172.21.15.23:0/881994789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec005040 con 0x7fd7fc060810 2021-11-19T08:39:54.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd7fb7fe700 1 --2- 172.21.15.23:0/881994789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7fc060810 0x7fd7fc104e80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fd7ec009830 tx=0x7fd7ec009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:54.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd7f8ff9700 1 -- 172.21.15.23:0/881994789 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7ec018070 con 0x7fd7fc060810 2021-11-19T08:39:54.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.185+0000 7fd7f8ff9700 1 -- 172.21.15.23:0/881994789 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7ec013410 con 0x7fd7fc060810 2021-11-19T08:39:54.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.186+0000 7fd801cc9700 1 -- 172.21.15.23:0/881994789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7fc1063c0 con 0x7fd7fc060810 2021-11-19T08:39:54.178 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.186+0000 7fd7f8ff9700 1 -- 172.21.15.23:0/881994789 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7ec01c620 con 0x7fd7fc060810 2021-11-19T08:39:54.178 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.186+0000 7fd801cc9700 1 -- 172.21.15.23:0/881994789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7fc106920 con 0x7fd7fc060810 2021-11-19T08:39:54.179 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.186+0000 7fd801cc9700 1 -- 172.21.15.23:0/881994789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7fc045ad0 con 0x7fd7fc060810 2021-11-19T08:39:54.179 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.187+0000 7fd7f8ff9700 1 -- 172.21.15.23:0/881994789 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd7ec01c8e0 con 0x7fd7fc060810 2021-11-19T08:39:54.180 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.187+0000 7fd7f8ff9700 1 --2- 172.21.15.23:0/881994789 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd7e805c2a0 0x7fd7e805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:54.180 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.187+0000 7fd7f8ff9700 1 -- 172.21.15.23:0/881994789 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fd7ec077b40 con 0x7fd7fc060810 2021-11-19T08:39:54.180 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.187+0000 7fd7f3fff700 1 --2- 172.21.15.23:0/881994789 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd7e805c2a0 0x7fd7e805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:54.180 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.188+0000 7fd7f3fff700 1 --2- 172.21.15.23:0/881994789 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd7e805c2a0 0x7fd7e805e760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fd7e40041d0 tx=0x7fd7e4009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:54.181 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.191+0000 7fd7f8ff9700 1 -- 172.21.15.23:0/881994789 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7ec04aa20 con 0x7fd7fc060810 2021-11-19T08:39:54.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.349+0000 7fd801cc9700 1 -- 172.21.15.23:0/881994789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd7fc0fef90 con 0x7fd7fc060810 2021-11-19T08:39:54.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.349+0000 7fd7f8ff9700 1 -- 172.21.15.23:0/881994789 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd7ec04e040 con 0x7fd7fc060810 2021-11-19T08:39:54.336 INFO:teuthology.orchestra.run.smithi023.stdout:77309411347 2021-11-19T08:39:54.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.351+0000 7fd801cc9700 1 -- 172.21.15.23:0/881994789 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd7e805c2a0 msgr2=0x7fd7e805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:54.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.351+0000 7fd801cc9700 1 --2- 172.21.15.23:0/881994789 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd7e805c2a0 0x7fd7e805e760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fd7e40041d0 tx=0x7fd7e4009040).stop 2021-11-19T08:39:54.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.351+0000 7fd801cc9700 1 -- 172.21.15.23:0/881994789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7fc060810 msgr2=0x7fd7fc104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:54.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.351+0000 7fd801cc9700 1 --2- 172.21.15.23:0/881994789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7fc060810 0x7fd7fc104e80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fd7ec009830 tx=0x7fd7ec009b30).stop 2021-11-19T08:39:54.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.351+0000 7fd801cc9700 1 -- 172.21.15.23:0/881994789 shutdown_connections 2021-11-19T08:39:54.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.351+0000 7fd801cc9700 1 --2- 172.21.15.23:0/881994789 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd7e805c2a0 0x7fd7e805e760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:54.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.351+0000 7fd801cc9700 1 --2- 172.21.15.23:0/881994789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7fc060810 0x7fd7fc104e80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:54.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.351+0000 7fd801cc9700 1 --2- 172.21.15.23:0/881994789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7fc0f99a0 0x7fd7fc105370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:54.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.351+0000 7fd801cc9700 1 -- 172.21.15.23:0/881994789 >> 172.21.15.23:0/881994789 conn(0x7fd7fc0f0e20 msgr2=0x7fd7fc0f52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:54.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.351+0000 7fd801cc9700 1 -- 172.21.15.23:0/881994789 shutdown_connections 2021-11-19T08:39:54.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:54.351+0000 7fd801cc9700 1 -- 172.21.15.23:0/881994789 wait complete. 2021-11-19T08:39:54.724 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411345 got 77309411347 for osd.2 2021-11-19T08:39:54.725 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.3 2021-11-19T08:39:54.751 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:54 smithi023 conmon[33490]: cluster 2021-11-19T08:39:52.819460+0000 mgr.smithi023.ehlpcl (mgr.14164) 163 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:54.752 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:54 smithi023 conmon[33490]: audit 2021-11-19T08:39:54.350510+0000 mon.smithi023 (mon.0) 441 : audit [DBG] from='client.? 172.21.15.23:0/881994789' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-11-19T08:39:55.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:54 smithi134 conmon[36363]: cluster 2021-11-19T08:39:52.819460+0000 mgr.smithi023.ehlpcl (mgr.14164) 163 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:55.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:54 smithi134 conmon[36363]: 2021-11-19T08:39:55.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:54 smithi134 conmon[36363]: audit 2021-11-19T08:39:54.350510+0000 mon.smithi023 (mon.0) 441 : audit [DBG] from='client.? 172.21.15.23:0/881994789' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-11-19T08:39:55.169 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:56.287 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.299+0000 7ff2388b2700 1 -- 172.21.15.23:0/1216549527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2340f58b0 msgr2=0x7ff2340f5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:56.287 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.299+0000 7ff2388b2700 1 --2- 172.21.15.23:0/1216549527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2340f58b0 0x7ff2340f5d30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7ff228004660 tx=0x7ff228009b30).stop 2021-11-19T08:39:56.287 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.300+0000 7ff2388b2700 1 -- 172.21.15.23:0/1216549527 shutdown_connections 2021-11-19T08:39:56.288 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.300+0000 7ff2388b2700 1 --2- 172.21.15.23:0/1216549527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2340f58b0 0x7ff2340f5d30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:56.288 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.300+0000 7ff2388b2700 1 --2- 172.21.15.23:0/1216549527 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2340f6ba0 0x7ff2340f6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:56.288 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.300+0000 7ff2388b2700 1 -- 172.21.15.23:0/1216549527 >> 172.21.15.23:0/1216549527 conn(0x7ff2340f0400 msgr2=0x7ff2340f2820 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:56.288 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.300+0000 7ff2388b2700 1 -- 172.21.15.23:0/1216549527 shutdown_connections 2021-11-19T08:39:56.288 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.300+0000 7ff2388b2700 1 -- 172.21.15.23:0/1216549527 wait complete. 2021-11-19T08:39:56.289 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.300+0000 7ff2388b2700 1 Processor -- start 2021-11-19T08:39:56.289 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.300+0000 7ff2388b2700 1 -- start start 2021-11-19T08:39:56.289 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff2388b2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2340f58b0 0x7ff234105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:56.289 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff2388b2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2340f6ba0 0x7ff234106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:56.290 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff2388b2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff234106890 con 0x7ff2340f6ba0 2021-11-19T08:39:56.290 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff2388b2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2341070d0 con 0x7ff2340f58b0 2021-11-19T08:39:56.290 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff23259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2340f6ba0 0x7ff234106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:56.290 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff23259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2340f6ba0 0x7ff234106290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46798/0 (socket says 172.21.15.23:46798) 2021-11-19T08:39:56.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff23259c700 1 -- 172.21.15.23:0/2251066040 learned_addr learned my addr 172.21.15.23:0/2251066040 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:56.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff232d9d700 1 --2- 172.21.15.23:0/2251066040 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2340f58b0 0x7ff234105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:56.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff23259c700 1 -- 172.21.15.23:0/2251066040 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2340f58b0 msgr2=0x7ff234105da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:56.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff23259c700 1 --2- 172.21.15.23:0/2251066040 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2340f58b0 0x7ff234105da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:56.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff23259c700 1 -- 172.21.15.23:0/2251066040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff228005040 con 0x7ff2340f6ba0 2021-11-19T08:39:56.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff23259c700 1 --2- 172.21.15.23:0/2251066040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2340f6ba0 0x7ff234106290 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7ff228009f50 tx=0x7ff228009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:56.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff21b7fe700 1 -- 172.21.15.23:0/2251066040 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff228018070 con 0x7ff2340f6ba0 2021-11-19T08:39:56.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff21b7fe700 1 -- 172.21.15.23:0/2251066040 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff228013410 con 0x7ff2340f6ba0 2021-11-19T08:39:56.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.301+0000 7ff2388b2700 1 -- 172.21.15.23:0/2251066040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff234107350 con 0x7ff2340f6ba0 2021-11-19T08:39:56.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.302+0000 7ff21b7fe700 1 -- 172.21.15.23:0/2251066040 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff22801c3c0 con 0x7ff2340f6ba0 2021-11-19T08:39:56.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.302+0000 7ff2388b2700 1 -- 172.21.15.23:0/2251066040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff234107830 con 0x7ff2340f6ba0 2021-11-19T08:39:56.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.303+0000 7ff21b7fe700 1 -- 172.21.15.23:0/2251066040 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff22801d420 con 0x7ff2340f6ba0 2021-11-19T08:39:56.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.303+0000 7ff2388b2700 1 -- 172.21.15.23:0/2251066040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff234059100 con 0x7ff2340f6ba0 2021-11-19T08:39:56.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.303+0000 7ff21b7fe700 1 --2- 172.21.15.23:0/2251066040 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff21c05c2b0 0x7ff21c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:56.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.303+0000 7ff21b7fe700 1 -- 172.21.15.23:0/2251066040 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff2280778a0 con 0x7ff2340f6ba0 2021-11-19T08:39:56.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.303+0000 7ff232d9d700 1 --2- 172.21.15.23:0/2251066040 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff21c05c2b0 0x7ff21c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:56.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.304+0000 7ff232d9d700 1 --2- 172.21.15.23:0/2251066040 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff21c05c2b0 0x7ff21c05e770 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7ff234106b60 tx=0x7ff224009c60).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:56.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.307+0000 7ff21b7fe700 1 -- 172.21.15.23:0/2251066040 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff22804a8a0 con 0x7ff2340f6ba0 2021-11-19T08:39:56.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.467+0000 7ff2388b2700 1 -- 172.21.15.23:0/2251066040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff2340008d0 con 0x7ff2340f6ba0 2021-11-19T08:39:56.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.467+0000 7ff21b7fe700 1 -- 172.21.15.23:0/2251066040 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff22804dec0 con 0x7ff2340f6ba0 2021-11-19T08:39:56.453 INFO:teuthology.orchestra.run.smithi023.stdout:111669149712 2021-11-19T08:39:56.454 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.469+0000 7ff2388b2700 1 -- 172.21.15.23:0/2251066040 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff21c05c2b0 msgr2=0x7ff21c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:56.454 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.469+0000 7ff2388b2700 1 --2- 172.21.15.23:0/2251066040 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff21c05c2b0 0x7ff21c05e770 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7ff234106b60 tx=0x7ff224009c60).stop 2021-11-19T08:39:56.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.469+0000 7ff2388b2700 1 -- 172.21.15.23:0/2251066040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2340f6ba0 msgr2=0x7ff234106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:56.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.469+0000 7ff2388b2700 1 --2- 172.21.15.23:0/2251066040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2340f6ba0 0x7ff234106290 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7ff228009f50 tx=0x7ff228009b30).stop 2021-11-19T08:39:56.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.469+0000 7ff2388b2700 1 -- 172.21.15.23:0/2251066040 shutdown_connections 2021-11-19T08:39:56.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.469+0000 7ff2388b2700 1 --2- 172.21.15.23:0/2251066040 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2340f58b0 0x7ff234105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:56.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.469+0000 7ff2388b2700 1 --2- 172.21.15.23:0/2251066040 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff21c05c2b0 0x7ff21c05e770 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:56.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.469+0000 7ff2388b2700 1 --2- 172.21.15.23:0/2251066040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2340f6ba0 0x7ff234106290 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:56.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.469+0000 7ff2388b2700 1 -- 172.21.15.23:0/2251066040 >> 172.21.15.23:0/2251066040 conn(0x7ff2340f0400 msgr2=0x7ff2340fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:56.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.469+0000 7ff2388b2700 1 -- 172.21.15.23:0/2251066040 shutdown_connections 2021-11-19T08:39:56.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:56.469+0000 7ff2388b2700 1 -- 172.21.15.23:0/2251066040 wait complete. 2021-11-19T08:39:56.751 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:56 smithi023 conmon[33490]: cluster 2021-11-19T08:39:54.819784+0000 2021-11-19T08:39:56.751 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:56 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 164 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:56.752 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:56 smithi023 conmon[33490]: audit 2021-11-19T08:39:56.468469+0000 mon.smithi023 (mon.0) 442 : audit [DBG] from='client.? 172.21.15.23:0/2251066040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-11-19T08:39:56.971 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149710 got 111669149712 for osd.3 2021-11-19T08:39:56.972 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.4 2021-11-19T08:39:57.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:56 smithi134 conmon[36363]: cluster 2021-11-19T08:39:54.819784+0000 2021-11-19T08:39:57.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:56 smithi134 conmon[36363]: mgr.smithi023.ehlpcl (mgr.14164) 164 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:57.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:56 smithi134 conmon[36363]: audit 2021-11-19T08:39:56.468469+0000 2021-11-19T08:39:57.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:56 smithi134 conmon[36363]: mon.smithi023 (mon.0) 442 : audit [DBG] from='client.? 172.21.15.23:0/2251066040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-11-19T08:39:57.458 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:58.562 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.575+0000 7ff37c8e5700 1 -- 172.21.15.23:0/1971587529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3780f7ce0 msgr2=0x7ff3780f8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:58.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.575+0000 7ff37c8e5700 1 --2- 172.21.15.23:0/1971587529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3780f7ce0 0x7ff3780f8100 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7ff368004660 tx=0x7ff368009b30).stop 2021-11-19T08:39:58.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.575+0000 7ff37c8e5700 1 -- 172.21.15.23:0/1971587529 shutdown_connections 2021-11-19T08:39:58.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.575+0000 7ff37c8e5700 1 --2- 172.21.15.23:0/1971587529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3780f62f0 0x7ff3780f6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:58.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.575+0000 7ff37c8e5700 1 --2- 172.21.15.23:0/1971587529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3780f7ce0 0x7ff3780f8100 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:58.564 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.575+0000 7ff37c8e5700 1 -- 172.21.15.23:0/1971587529 >> 172.21.15.23:0/1971587529 conn(0x7ff3780f1d70 msgr2=0x7ff3780f41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:58.564 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.575+0000 7ff37c8e5700 1 -- 172.21.15.23:0/1971587529 shutdown_connections 2021-11-19T08:39:58.564 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.575+0000 7ff37c8e5700 1 -- 172.21.15.23:0/1971587529 wait complete. 2021-11-19T08:39:58.564 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.576+0000 7ff37c8e5700 1 Processor -- start 2021-11-19T08:39:58.564 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.576+0000 7ff37c8e5700 1 -- start start 2021-11-19T08:39:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.576+0000 7ff37c8e5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3780f62f0 0x7ff378106fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.576+0000 7ff37c8e5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3780f7ce0 0x7ff3781074c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.576+0000 7ff37c8e5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff378107a70 con 0x7ff3780f7ce0 2021-11-19T08:39:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.576+0000 7ff37c8e5700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff378107bb0 con 0x7ff3780f62f0 2021-11-19T08:39:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.576+0000 7ff37659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3780f7ce0 0x7ff3781074c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:58.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.576+0000 7ff37659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3780f7ce0 0x7ff3781074c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46808/0 (socket says 172.21.15.23:46808) 2021-11-19T08:39:58.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.576+0000 7ff37659c700 1 -- 172.21.15.23:0/3613943861 learned_addr learned my addr 172.21.15.23:0/3613943861 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:39:58.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.576+0000 7ff376d9d700 1 --2- 172.21.15.23:0/3613943861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3780f62f0 0x7ff378106fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:58.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.577+0000 7ff37659c700 1 -- 172.21.15.23:0/3613943861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3780f62f0 msgr2=0x7ff378106fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:58.567 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.577+0000 7ff37659c700 1 --2- 172.21.15.23:0/3613943861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3780f62f0 0x7ff378106fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:58.567 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.577+0000 7ff37659c700 1 -- 172.21.15.23:0/3613943861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff368005040 con 0x7ff3780f7ce0 2021-11-19T08:39:58.567 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.577+0000 7ff37659c700 1 --2- 172.21.15.23:0/3613943861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3780f7ce0 0x7ff3781074c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7ff36c0026d0 tx=0x7ff36c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:58.567 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.577+0000 7ff35f7fe700 1 -- 172.21.15.23:0/3613943861 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff36c0079c0 con 0x7ff3780f7ce0 2021-11-19T08:39:58.568 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.577+0000 7ff35f7fe700 1 -- 172.21.15.23:0/3613943861 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff36c00f040 con 0x7ff3780f7ce0 2021-11-19T08:39:58.568 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.577+0000 7ff35f7fe700 1 -- 172.21.15.23:0/3613943861 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff36c0148b0 con 0x7ff3780f7ce0 2021-11-19T08:39:58.568 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.577+0000 7ff37c8e5700 1 -- 172.21.15.23:0/3613943861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff378108550 con 0x7ff3780f7ce0 2021-11-19T08:39:58.568 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.577+0000 7ff37c8e5700 1 -- 172.21.15.23:0/3613943861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff378108b10 con 0x7ff3780f7ce0 2021-11-19T08:39:58.569 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.578+0000 7ff37c8e5700 1 -- 172.21.15.23:0/3613943861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff358004fa0 con 0x7ff3780f7ce0 2021-11-19T08:39:58.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.579+0000 7ff35f7fe700 1 -- 172.21.15.23:0/3613943861 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff36c014a10 con 0x7ff3780f7ce0 2021-11-19T08:39:58.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.579+0000 7ff35f7fe700 1 --2- 172.21.15.23:0/3613943861 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff36005c2b0 0x7ff36005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:39:58.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.579+0000 7ff35f7fe700 1 -- 172.21.15.23:0/3613943861 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff36c0759e0 con 0x7ff3780f7ce0 2021-11-19T08:39:58.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.579+0000 7ff376d9d700 1 --2- 172.21.15.23:0/3613943861 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff36005c2b0 0x7ff36005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:39:58.571 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.579+0000 7ff376d9d700 1 --2- 172.21.15.23:0/3613943861 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff36005c2b0 0x7ff36005e770 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7ff368009f50 tx=0x7ff368009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:39:58.571 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.583+0000 7ff35f7fe700 1 -- 172.21.15.23:0/3613943861 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff36c048a60 con 0x7ff3780f7ce0 2021-11-19T08:39:58.729 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.743+0000 7ff37c8e5700 1 -- 172.21.15.23:0/3613943861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff358005670 con 0x7ff3780f7ce0 2021-11-19T08:39:58.729 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.744+0000 7ff35f7fe700 1 -- 172.21.15.23:0/3613943861 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff36c04c080 con 0x7ff3780f7ce0 2021-11-19T08:39:58.729 INFO:teuthology.orchestra.run.smithi023.stdout:133143986189 2021-11-19T08:39:58.730 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.745+0000 7ff37c8e5700 1 -- 172.21.15.23:0/3613943861 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff36005c2b0 msgr2=0x7ff36005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.745+0000 7ff37c8e5700 1 --2- 172.21.15.23:0/3613943861 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff36005c2b0 0x7ff36005e770 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7ff368009f50 tx=0x7ff368009b30).stop 2021-11-19T08:39:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.745+0000 7ff37c8e5700 1 -- 172.21.15.23:0/3613943861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3780f7ce0 msgr2=0x7ff3781074c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:39:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.745+0000 7ff37c8e5700 1 --2- 172.21.15.23:0/3613943861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3780f7ce0 0x7ff3781074c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7ff36c0026d0 tx=0x7ff36c009de0).stop 2021-11-19T08:39:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.745+0000 7ff37c8e5700 1 -- 172.21.15.23:0/3613943861 shutdown_connections 2021-11-19T08:39:58.732 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.745+0000 7ff37c8e5700 1 --2- 172.21.15.23:0/3613943861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3780f62f0 0x7ff378106fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:58.732 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.745+0000 7ff37c8e5700 1 --2- 172.21.15.23:0/3613943861 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff36005c2b0 0x7ff36005e770 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:58.732 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.745+0000 7ff37c8e5700 1 --2- 172.21.15.23:0/3613943861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3780f7ce0 0x7ff3781074c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:39:58.732 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.745+0000 7ff37c8e5700 1 -- 172.21.15.23:0/3613943861 >> 172.21.15.23:0/3613943861 conn(0x7ff3780f1d70 msgr2=0x7ff3780fb320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:39:58.733 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.745+0000 7ff37c8e5700 1 -- 172.21.15.23:0/3613943861 shutdown_connections 2021-11-19T08:39:58.733 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:39:58.745+0000 7ff37c8e5700 1 -- 172.21.15.23:0/3613943861 wait complete. 2021-11-19T08:39:58.851 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:58 smithi023 conmon[33490]: cluster 2021-11-19T08:39:56.820106+0000 mgr.smithi023.ehlpcl (mgr.14164) 165 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:59.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:58 smithi134 conmon[36363]: cluster 2021-11-19T08:39:56.820106+0000 mgr.smithi023.ehlpcl (mgr.14164) 165 : cluster 2021-11-19T08:39:59.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:58 smithi134 conmon[36363]: [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:39:59.072 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986187 got 133143986189 for osd.4 2021-11-19T08:39:59.073 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph osd last-stat-seq osd.5 2021-11-19T08:39:59.516 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:39:59.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:59 smithi023 conmon[33490]: audit 2021-11-19T08:39:58.744931+0000 2021-11-19T08:39:59.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:39:59 smithi023 conmon[33490]: mon.smithi023 (mon.0) 443 : audit [DBG] from='client.? 172.21.15.23:0/3613943861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-11-19T08:40:00.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:59 smithi134 conmon[36363]: audit 2021-11-19T08:39:58.744931+0000 mon.smithi023 (mon.0 2021-11-19T08:40:00.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:39:59 smithi134 conmon[36363]: ) 443 : audit [DBG] from='client.? 172.21.15.23:0/3613943861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-11-19T08:40:00.553 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.566+0000 7f038a642700 1 -- 172.21.15.23:0/3321044383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03840f67e0 msgr2=0x7f03840f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:00.553 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.566+0000 7f038a642700 1 --2- 172.21.15.23:0/3321044383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03840f67e0 0x7f03840f6c00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f0374004660 tx=0x7f0374009b30).stop 2021-11-19T08:40:00.554 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.566+0000 7f038a642700 1 -- 172.21.15.23:0/3321044383 shutdown_connections 2021-11-19T08:40:00.554 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.566+0000 7f038a642700 1 --2- 172.21.15.23:0/3321044383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03840f7a80 0x7f03840f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:00.554 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.566+0000 7f038a642700 1 --2- 172.21.15.23:0/3321044383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03840f67e0 0x7f03840f6c00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:00.554 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.566+0000 7f038a642700 1 -- 172.21.15.23:0/3321044383 >> 172.21.15.23:0/3321044383 conn(0x7f03840f1f20 msgr2=0x7f03840f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:00.554 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.566+0000 7f038a642700 1 -- 172.21.15.23:0/3321044383 shutdown_connections 2021-11-19T08:40:00.555 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.566+0000 7f038a642700 1 -- 172.21.15.23:0/3321044383 wait complete. 2021-11-19T08:40:00.555 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.566+0000 7f038a642700 1 Processor -- start 2021-11-19T08:40:00.555 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.567+0000 7f038a642700 1 -- start start 2021-11-19T08:40:00.555 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.567+0000 7f038a642700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03840f67e0 0x7f038406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:00.556 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.567+0000 7f038a642700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03840f7a80 0x7f0384069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:00.556 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.567+0000 7f038a642700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038410ad60 con 0x7f03840f7a80 2021-11-19T08:40:00.556 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.567+0000 7f038a642700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038410aea0 con 0x7f03840f67e0 2021-11-19T08:40:00.556 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.567+0000 7f03837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03840f7a80 0x7f0384069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:00.556 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.567+0000 7f03837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03840f7a80 0x7f0384069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46816/0 (socket says 172.21.15.23:46816) 2021-11-19T08:40:00.557 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.567+0000 7f03837fe700 1 -- 172.21.15.23:0/1137113400 learned_addr learned my addr 172.21.15.23:0/1137113400 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:00.557 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.567+0000 7f0383fff700 1 --2- 172.21.15.23:0/1137113400 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03840f67e0 0x7f038406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:00.557 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.568+0000 7f03837fe700 1 -- 172.21.15.23:0/1137113400 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03840f67e0 msgr2=0x7f038406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:00.557 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.568+0000 7f03837fe700 1 --2- 172.21.15.23:0/1137113400 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03840f67e0 0x7f038406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:00.558 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.568+0000 7f03837fe700 1 -- 172.21.15.23:0/1137113400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0374005040 con 0x7f03840f7a80 2021-11-19T08:40:00.558 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.568+0000 7f03837fe700 1 --2- 172.21.15.23:0/1137113400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03840f7a80 0x7f0384069e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f036c00c530 tx=0x7f036c007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:00.558 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.568+0000 7f0380ff9700 1 -- 172.21.15.23:0/1137113400 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f036c00f650 con 0x7f03840f7a80 2021-11-19T08:40:00.559 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.568+0000 7f038a642700 1 -- 172.21.15.23:0/1137113400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f038410b180 con 0x7f03840f7a80 2021-11-19T08:40:00.559 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.568+0000 7f0380ff9700 1 -- 172.21.15.23:0/1137113400 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f036c00fc60 con 0x7f03840f7a80 2021-11-19T08:40:00.560 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.568+0000 7f0380ff9700 1 -- 172.21.15.23:0/1137113400 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f036c012c30 con 0x7f03840f7a80 2021-11-19T08:40:00.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.568+0000 7f038a642700 1 -- 172.21.15.23:0/1137113400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038410b740 con 0x7f03840f7a80 2021-11-19T08:40:00.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.569+0000 7f038a642700 1 -- 172.21.15.23:0/1137113400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0384045ad0 con 0x7f03840f7a80 2021-11-19T08:40:00.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.569+0000 7f0380ff9700 1 -- 172.21.15.23:0/1137113400 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f036c010070 con 0x7f03840f7a80 2021-11-19T08:40:00.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.570+0000 7f0380ff9700 1 --2- 172.21.15.23:0/1137113400 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f036405c250 0x7f036405e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:00.562 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.570+0000 7f0380ff9700 1 -- 172.21.15.23:0/1137113400 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f036c0757f0 con 0x7f03840f7a80 2021-11-19T08:40:00.562 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.570+0000 7f0383fff700 1 --2- 172.21.15.23:0/1137113400 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f036405c250 0x7f036405e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:00.562 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.570+0000 7f0383fff700 1 --2- 172.21.15.23:0/1137113400 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f036405c250 0x7f036405e710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f038406b730 tx=0x7f0374009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:00.562 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.574+0000 7f0380ff9700 1 -- 172.21.15.23:0/1137113400 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f036c0452b0 con 0x7f03840f7a80 2021-11-19T08:40:00.718 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.733+0000 7f038a642700 1 -- 172.21.15.23:0/1137113400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f03840fef70 con 0x7f03840f7a80 2021-11-19T08:40:00.719 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.733+0000 7f0380ff9700 1 -- 172.21.15.23:0/1137113400 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f036c0487f0 con 0x7f03840f7a80 2021-11-19T08:40:00.719 INFO:teuthology.orchestra.run.smithi023.stdout:150323855371 2021-11-19T08:40:00.720 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.735+0000 7f038a642700 1 -- 172.21.15.23:0/1137113400 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f036405c250 msgr2=0x7f036405e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:00.721 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.735+0000 7f038a642700 1 --2- 172.21.15.23:0/1137113400 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f036405c250 0x7f036405e710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f038406b730 tx=0x7f0374009b30).stop 2021-11-19T08:40:00.721 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.735+0000 7f038a642700 1 -- 172.21.15.23:0/1137113400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03840f7a80 msgr2=0x7f0384069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:00.721 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.735+0000 7f038a642700 1 --2- 172.21.15.23:0/1137113400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03840f7a80 0x7f0384069e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f036c00c530 tx=0x7f036c007e20).stop 2021-11-19T08:40:00.721 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.735+0000 7f038a642700 1 -- 172.21.15.23:0/1137113400 shutdown_connections 2021-11-19T08:40:00.722 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.735+0000 7f038a642700 1 --2- 172.21.15.23:0/1137113400 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03840f67e0 0x7f038406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:00.722 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.735+0000 7f038a642700 1 --2- 172.21.15.23:0/1137113400 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f036405c250 0x7f036405e710 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:00.722 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.735+0000 7f038a642700 1 --2- 172.21.15.23:0/1137113400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03840f7a80 0x7f0384069e20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:00.722 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.735+0000 7f038a642700 1 -- 172.21.15.23:0/1137113400 >> 172.21.15.23:0/1137113400 conn(0x7f03840f1f20 msgr2=0x7f03840fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:00.723 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.735+0000 7f038a642700 1 -- 172.21.15.23:0/1137113400 shutdown_connections 2021-11-19T08:40:00.723 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:00.735+0000 7f038a642700 1 -- 172.21.15.23:0/1137113400 wait complete. 2021-11-19T08:40:00.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:00 smithi023 conmon[33490]: cluster 2021-11-19T08:39:58.820397+0000 mgr.smithi023.ehlpcl (mgr.14164) 166 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:00.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:00 smithi023 conmon[33490]: cluster 2021-11-19T08:40:00.000104+0000 mon.smithi023 (mon.0) 444 : cluster [INF] overall HEALTH_OK 2021-11-19T08:40:01.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:00 smithi134 conmon[36363]: cluster 2021-11-19T08:39:58.820397+0000 mgr.smithi023.ehlpcl 2021-11-19T08:40:01.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:00 smithi134 conmon[36363]: (mgr.14164) 166 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:01.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:00 smithi134 conmon[36363]: cluster 2021-11-19T08:40:00.000104+0000 mon.smithi023 (mon.0) 444 : cluster 2021-11-19T08:40:01.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:00 smithi134 conmon[36363]: [INF] overall HEALTH_OK 2021-11-19T08:40:01.131 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855369 got 150323855371 for osd.5 2021-11-19T08:40:01.131 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2021-11-19T08:40:01.132 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph pg dump --format=json 2021-11-19T08:40:01.576 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:40:01.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:01 smithi023 conmon[33490]: audit 2021-11-19T08:40:00.734566+0000 2021-11-19T08:40:01.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:01 smithi023 conmon[33490]: mon.smithi023 (mon.0) 445 : audit [DBG] from='client.? 172.21.15.23:0/1137113400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-11-19T08:40:02.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:01 smithi134 conmon[36363]: audit 2021-11-19T08:40:00.734566+0000 mon.smithi023 (mon.0) 445 : audit [DBG] from='client.? 172.21.15.23:0/1137113400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-11-19T08:40:02.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:01 smithi134 conmon[36363]: 2021-11-19T08:40:02.763 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.776+0000 7fbb4a89d700 1 -- 172.21.15.23:0/1174432015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb440f62f0 msgr2=0x7fbb440f6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:02.764 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.776+0000 7fbb4a89d700 1 --2- 172.21.15.23:0/1174432015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb440f62f0 0x7fbb440f6750 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fbb34004660 tx=0x7fbb34009b30).stop 2021-11-19T08:40:02.764 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.777+0000 7fbb4a89d700 1 -- 172.21.15.23:0/1174432015 shutdown_connections 2021-11-19T08:40:02.764 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.777+0000 7fbb4a89d700 1 --2- 172.21.15.23:0/1174432015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb440f62f0 0x7fbb440f6750 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:02.764 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.777+0000 7fbb4a89d700 1 --2- 172.21.15.23:0/1174432015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb440f7ce0 0x7fbb440f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:02.765 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.777+0000 7fbb4a89d700 1 -- 172.21.15.23:0/1174432015 >> 172.21.15.23:0/1174432015 conn(0x7fbb440f1d70 msgr2=0x7fbb440f41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:02.765 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.777+0000 7fbb4a89d700 1 -- 172.21.15.23:0/1174432015 shutdown_connections 2021-11-19T08:40:02.765 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.777+0000 7fbb4a89d700 1 -- 172.21.15.23:0/1174432015 wait complete. 2021-11-19T08:40:02.765 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.777+0000 7fbb4a89d700 1 Processor -- start 2021-11-19T08:40:02.766 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.777+0000 7fbb4a89d700 1 -- start start 2021-11-19T08:40:02.766 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.777+0000 7fbb4a89d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb440f62f0 0x7fbb4406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:02.767 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.777+0000 7fbb4a89d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb440f7ce0 0x7fbb44069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:02.768 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.777+0000 7fbb4a89d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4406a380 con 0x7fbb440f62f0 2021-11-19T08:40:02.768 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.777+0000 7fbb4a89d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4406a4c0 con 0x7fbb440f7ce0 2021-11-19T08:40:02.768 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.778+0000 7fbb4989b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb440f62f0 0x7fbb4406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:02.768 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.778+0000 7fbb4989b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb440f62f0 0x7fbb4406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46824/0 (socket says 172.21.15.23:46824) 2021-11-19T08:40:02.769 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.778+0000 7fbb4989b700 1 -- 172.21.15.23:0/3915767073 learned_addr learned my addr 172.21.15.23:0/3915767073 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:02.769 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.778+0000 7fbb4909a700 1 --2- 172.21.15.23:0/3915767073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb440f7ce0 0x7fbb44069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:02.769 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.778+0000 7fbb4989b700 1 -- 172.21.15.23:0/3915767073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb440f7ce0 msgr2=0x7fbb44069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:02.769 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.778+0000 7fbb4989b700 1 --2- 172.21.15.23:0/3915767073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb440f7ce0 0x7fbb44069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:02.769 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.778+0000 7fbb4989b700 1 -- 172.21.15.23:0/3915767073 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb34005040 con 0x7fbb440f62f0 2021-11-19T08:40:02.770 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.778+0000 7fbb4989b700 1 --2- 172.21.15.23:0/3915767073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb440f62f0 0x7fbb4406b9a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fbb40002700 tx=0x7fbb40009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:02.770 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.778+0000 7fbb3a7fc700 1 -- 172.21.15.23:0/3915767073 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb400079c0 con 0x7fbb440f62f0 2021-11-19T08:40:02.770 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.778+0000 7fbb3a7fc700 1 -- 172.21.15.23:0/3915767073 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb4000f040 con 0x7fbb440f62f0 2021-11-19T08:40:02.770 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.778+0000 7fbb4a89d700 1 -- 172.21.15.23:0/3915767073 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb4406a7a0 con 0x7fbb440f62f0 2021-11-19T08:40:02.771 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.778+0000 7fbb3a7fc700 1 -- 172.21.15.23:0/3915767073 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb40014860 con 0x7fbb440f62f0 2021-11-19T08:40:02.771 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.778+0000 7fbb4a89d700 1 -- 172.21.15.23:0/3915767073 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4410efd0 con 0x7fbb440f62f0 2021-11-19T08:40:02.772 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.779+0000 7fbb3a7fc700 1 -- 172.21.15.23:0/3915767073 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbb4001b060 con 0x7fbb440f62f0 2021-11-19T08:40:02.772 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.779+0000 7fbb4a89d700 1 -- 172.21.15.23:0/3915767073 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb44045ad0 con 0x7fbb440f62f0 2021-11-19T08:40:02.773 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.780+0000 7fbb3a7fc700 1 --2- 172.21.15.23:0/3915767073 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbb3005c2b0 0x7fbb3005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:02.773 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.780+0000 7fbb3a7fc700 1 -- 172.21.15.23:0/3915767073 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fbb40075990 con 0x7fbb440f62f0 2021-11-19T08:40:02.773 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.780+0000 7fbb4909a700 1 --2- 172.21.15.23:0/3915767073 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbb3005c2b0 0x7fbb3005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:02.773 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.780+0000 7fbb4909a700 1 --2- 172.21.15.23:0/3915767073 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbb3005c2b0 0x7fbb3005e770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fbb34009f50 tx=0x7fbb34009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:02.774 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.784+0000 7fbb3a7fc700 1 -- 172.21.15.23:0/3915767073 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb40048a10 con 0x7fbb440f62f0 2021-11-19T08:40:02.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:02 smithi023 conmon[33490]: cluster 2021-11-19T08:40:00.820743+0000 mgr.smithi023.ehlpcl (mgr.14164) 167 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:02.924 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.939+0000 7fbb4a89d700 1 -- 172.21.15.23:0/3915767073 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbb440fef10 con 0x7fbb3005c2b0 2021-11-19T08:40:02.925 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.940+0000 7fbb3a7fc700 1 -- 172.21.15.23:0/3915767073 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24036 (secure 0 0 0) 0x7fbb440fef10 con 0x7fbb3005c2b0 2021-11-19T08:40:02.926 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:02.927 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.942+0000 7fbb4a89d700 1 -- 172.21.15.23:0/3915767073 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbb3005c2b0 msgr2=0x7fbb3005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:02.927 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.942+0000 7fbb4a89d700 1 --2- 172.21.15.23:0/3915767073 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbb3005c2b0 0x7fbb3005e770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fbb34009f50 tx=0x7fbb34009b30).stop 2021-11-19T08:40:02.928 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.942+0000 7fbb4a89d700 1 -- 172.21.15.23:0/3915767073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb440f62f0 msgr2=0x7fbb4406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:02.928 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.942+0000 7fbb4a89d700 1 --2- 172.21.15.23:0/3915767073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb440f62f0 0x7fbb4406b9a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fbb40002700 tx=0x7fbb40009d80).stop 2021-11-19T08:40:02.928 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.942+0000 7fbb4a89d700 1 -- 172.21.15.23:0/3915767073 shutdown_connections 2021-11-19T08:40:02.928 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.942+0000 7fbb4a89d700 1 --2- 172.21.15.23:0/3915767073 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbb3005c2b0 0x7fbb3005e770 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:02.929 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.942+0000 7fbb4a89d700 1 --2- 172.21.15.23:0/3915767073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb440f62f0 0x7fbb4406b9a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:02.929 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.942+0000 7fbb4a89d700 1 --2- 172.21.15.23:0/3915767073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb440f7ce0 0x7fbb44069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:02.929 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.942+0000 7fbb4a89d700 1 -- 172.21.15.23:0/3915767073 >> 172.21.15.23:0/3915767073 conn(0x7fbb440f1d70 msgr2=0x7fbb440fb320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:02.929 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.942+0000 7fbb4a89d700 1 -- 172.21.15.23:0/3915767073 shutdown_connections 2021-11-19T08:40:02.929 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:02.942+0000 7fbb4a89d700 1 -- 172.21.15.23:0/3915767073 wait complete. 2021-11-19T08:40:02.930 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2021-11-19T08:40:03.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:02 smithi134 conmon[36363]: cluster 2021-11-19T08:40:00.820743+0000 mgr.smithi023.ehlpcl (mgr.14164) 167 : cluster [DBG] 2021-11-19T08:40:03.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:02 smithi134 conmon[36363]: pgmap v136: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:03.288 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2021-11-19T08:40:02.820929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":30936,"kb_used_data":1896,"kb_used_omap":0,"kb_used_meta":28992,"kb_avail":562439976,"statfs":{"total":575970213888,"available":575938535424,"internally_reserved":0,"allocated":1941504,"data_stored":722634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29687808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"33","reported_epoch":"35","state":"active+clean","last_fresh":"2021-11-19T08:39:20.517528+0000","last_change":"2021-11-19T08:39:04.385737+0000","last_active":"2021-11-19T08:39:20.517528+0000","last_peered":"2021-11-19T08:39:20.517528+0000","last_clean":"2021-11-19T08:39:20.517528+0000","last_became_active":"2021-11-19T08:39:04.385272+0000","last_became_peered":"2021-11-19T08:39:04.385272+0000","last_unstale":"2021-11-19T08:39:20.517528+0000","last_undegraded":"2021-11-19T08:39:20.517528+0000","last_fullsized":"2021-11-19T08:39:20.517528+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-11-19T08:38:43.221904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-11-19T08:38:43.221904+0000","last_clean_scrub_stamp":"2021-11-19T08:38:43.221904+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":35,"seq":150323855372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5060,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740092,"statfs":{"total":95995035648,"available":95989854208,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5060,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740092,"statfs":{"total":95995035648,"available":95989854208,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5124,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740028,"statfs":{"total":95995035648,"available":95989788672,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 19 08:39:49 2021","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.50600000000000001}]},{"osd":1,"last update":"Fri Nov 19 08:39:49 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Nov 19 08:39:49 2021","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.45500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411349,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 19 08:39:36 2021","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.435,"5min":0.435,"15min":0.435},"last":0.29299999999999998}]},{"osd":1,"last update":"Fri Nov 19 08:39:36 2021","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.309},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.245}]},{"osd":3,"last update":"Fri Nov 19 08:39:48 2021","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]}]},{"osd":0,"up_from":9,"seq":38654705692,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5252,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739900,"statfs":{"total":95995035648,"available":95989657600,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 19 08:39:15 2021","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.29799999999999999}]},{"osd":2,"last update":"Fri Nov 19 08:39:36 2021","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.315,"15min":0.315},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.317},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"last":0.34499999999999997}]},{"osd":3,"last update":"Fri Nov 19 08:39:52 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.58699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5252,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739900,"statfs":{"total":95995035648,"available":95989657600,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 19 08:39:17 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.44,"5min":0.44,"15min":0.44},"last":0.378}]},{"osd":2,"last update":"Fri Nov 19 08:39:35 2021","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.34100000000000003}]},{"osd":3,"last update":"Fri Nov 19 08:39:50 2021","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-11-19T08:40:03.290 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph pg dump --format=json 2021-11-19T08:40:03.732 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:40:04.879 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.893+0000 7f5bcb114700 1 -- 172.21.15.23:0/2167484221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bc40f56c0 msgr2=0x7f5bc40f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:04.879 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.893+0000 7f5bcb114700 1 --2- 172.21.15.23:0/2167484221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bc40f56c0 0x7f5bc40f5ae0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f5bbc004660 tx=0x7f5bbc009b30).stop 2021-11-19T08:40:04.880 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:04 smithi023 conmon[33490]: cluster 2021-11-19T08:40:02.821057+0000 2021-11-19T08:40:04.880 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:04 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 168 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:04.880 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:04 smithi023 conmon[33490]: audit 2021-11-19T08:40:02.940555+0000 mgr.smithi023.ehlpcl (mgr.14164) 169 : audit [DBG] from='client.14394 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-19T08:40:04.882 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.893+0000 7f5bcb114700 1 -- 172.21.15.23:0/2167484221 shutdown_connections 2021-11-19T08:40:04.883 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.893+0000 7f5bcb114700 1 --2- 172.21.15.23:0/2167484221 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bc40f6960 0x7f5bc40f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:04.883 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.893+0000 7f5bcb114700 1 --2- 172.21.15.23:0/2167484221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bc40f56c0 0x7f5bc40f5ae0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:04.883 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.893+0000 7f5bcb114700 1 -- 172.21.15.23:0/2167484221 >> 172.21.15.23:0/2167484221 conn(0x7f5bc40f0e20 msgr2=0x7f5bc40f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:04.883 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.893+0000 7f5bcb114700 1 -- 172.21.15.23:0/2167484221 shutdown_connections 2021-11-19T08:40:04.884 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.893+0000 7f5bcb114700 1 -- 172.21.15.23:0/2167484221 wait complete. 2021-11-19T08:40:04.884 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.894+0000 7f5bcb114700 1 Processor -- start 2021-11-19T08:40:04.884 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.894+0000 7f5bcb114700 1 -- start start 2021-11-19T08:40:04.884 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.894+0000 7f5bcb114700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bc40f56c0 0x7f5bc4109d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:04.884 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.894+0000 7f5bcb114700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bc40f6960 0x7f5bc410c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:04.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.894+0000 7f5bcb114700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc4060980 con 0x7f5bc40f56c0 2021-11-19T08:40:04.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.894+0000 7f5bcb114700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc4060ac0 con 0x7f5bc40f6960 2021-11-19T08:40:04.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.894+0000 7f5bc8eb0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bc40f56c0 0x7f5bc4109d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:04.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.894+0000 7f5bc8eb0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bc40f56c0 0x7f5bc4109d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46834/0 (socket says 172.21.15.23:46834) 2021-11-19T08:40:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.894+0000 7f5bc8eb0700 1 -- 172.21.15.23:0/1824710556 learned_addr learned my addr 172.21.15.23:0/1824710556 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.894+0000 7f5bc3fff700 1 --2- 172.21.15.23:0/1824710556 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bc40f6960 0x7f5bc410c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.895+0000 7f5bc8eb0700 1 -- 172.21.15.23:0/1824710556 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bc40f6960 msgr2=0x7f5bc410c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.895+0000 7f5bc8eb0700 1 --2- 172.21.15.23:0/1824710556 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bc40f6960 0x7f5bc410c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.895+0000 7f5bc8eb0700 1 -- 172.21.15.23:0/1824710556 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bbc005040 con 0x7f5bc40f56c0 2021-11-19T08:40:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.895+0000 7f5bc8eb0700 1 --2- 172.21.15.23:0/1824710556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bc40f56c0 0x7f5bc4109d90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f5bbc009f50 tx=0x7f5bbc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.895+0000 7f5bc17fa700 1 -- 172.21.15.23:0/1824710556 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bbc018070 con 0x7f5bc40f56c0 2021-11-19T08:40:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.895+0000 7f5bcb114700 1 -- 172.21.15.23:0/1824710556 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc410c840 con 0x7f5bc40f56c0 2021-11-19T08:40:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.895+0000 7f5bc17fa700 1 -- 172.21.15.23:0/1824710556 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5bbc013410 con 0x7f5bc40f56c0 2021-11-19T08:40:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.895+0000 7f5bc17fa700 1 -- 172.21.15.23:0/1824710556 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bbc01c3c0 con 0x7f5bc40f56c0 2021-11-19T08:40:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.895+0000 7f5bcb114700 1 -- 172.21.15.23:0/1824710556 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc410cd70 con 0x7f5bc40f56c0 2021-11-19T08:40:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.896+0000 7f5bcb114700 1 -- 172.21.15.23:0/1824710556 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc4045ad0 con 0x7f5bc40f56c0 2021-11-19T08:40:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.896+0000 7f5bc17fa700 1 -- 172.21.15.23:0/1824710556 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5bbc01d420 con 0x7f5bc40f56c0 2021-11-19T08:40:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.897+0000 7f5bc17fa700 1 --2- 172.21.15.23:0/1824710556 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5bb005c2a0 0x7f5bb005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.897+0000 7f5bc17fa700 1 -- 172.21.15.23:0/1824710556 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f5bbc0778f0 con 0x7f5bc40f56c0 2021-11-19T08:40:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.897+0000 7f5bc3fff700 1 --2- 172.21.15.23:0/1824710556 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5bb005c2a0 0x7f5bb005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.897+0000 7f5bc3fff700 1 --2- 172.21.15.23:0/1824710556 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5bb005c2a0 0x7f5bb005e760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5bb4004550 tx=0x7f5bb4009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:04.890 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:04.901+0000 7f5bc17fa700 1 -- 172.21.15.23:0/1824710556 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5bbc04a8f0 con 0x7f5bc40f56c0 2021-11-19T08:40:05.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:04 smithi134 conmon[36363]: cluster 2021-11-19T08:40:02.821057+0000 mgr.smithi023.ehlpcl (mgr.14164) 168 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:05.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:04 smithi134 conmon[36363]: audit 2021-11-19T08:40:02.940555+0000 mgr.smithi023.ehlpcl (mgr.14164) 169 : audit [DBG] from='client.14394 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-19T08:40:05.043 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:05.057+0000 7f5bcb114700 1 -- 172.21.15.23:0/1824710556 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5bc40fdf90 con 0x7f5bb005c2a0 2021-11-19T08:40:05.044 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:05.058+0000 7f5bc17fa700 1 -- 172.21.15.23:0/1824710556 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+24107 (secure 0 0 0) 0x7f5bc40fdf90 con 0x7f5bb005c2a0 2021-11-19T08:40:05.045 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:05.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:05.060+0000 7f5bcb114700 1 -- 172.21.15.23:0/1824710556 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5bb005c2a0 msgr2=0x7f5bb005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:05.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:05.060+0000 7f5bcb114700 1 --2- 172.21.15.23:0/1824710556 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5bb005c2a0 0x7f5bb005e760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5bb4004550 tx=0x7f5bb4009040).stop 2021-11-19T08:40:05.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:05.060+0000 7f5bcb114700 1 -- 172.21.15.23:0/1824710556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bc40f56c0 msgr2=0x7f5bc4109d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:05.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:05.060+0000 7f5bcb114700 1 --2- 172.21.15.23:0/1824710556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bc40f56c0 0x7f5bc4109d90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f5bbc009f50 tx=0x7f5bbc009b30).stop 2021-11-19T08:40:05.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:05.060+0000 7f5bcb114700 1 -- 172.21.15.23:0/1824710556 shutdown_connections 2021-11-19T08:40:05.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:05.060+0000 7f5bcb114700 1 --2- 172.21.15.23:0/1824710556 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5bb005c2a0 0x7f5bb005e760 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:05.050 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:05.060+0000 7f5bcb114700 1 --2- 172.21.15.23:0/1824710556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bc40f56c0 0x7f5bc4109d90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:05.050 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:05.060+0000 7f5bcb114700 1 --2- 172.21.15.23:0/1824710556 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5bc40f6960 0x7f5bc410c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:05.050 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:05.060+0000 7f5bcb114700 1 -- 172.21.15.23:0/1824710556 >> 172.21.15.23:0/1824710556 conn(0x7f5bc40f0e20 msgr2=0x7f5bc40fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:05.051 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:05.061+0000 7f5bcb114700 1 -- 172.21.15.23:0/1824710556 shutdown_connections 2021-11-19T08:40:05.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:05.061+0000 7f5bcb114700 1 -- 172.21.15.23:0/1824710556 wait complete. 2021-11-19T08:40:05.053 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2021-11-19T08:40:05.536 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2021-11-19T08:40:04.821202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":30936,"kb_used_data":1896,"kb_used_omap":0,"kb_used_meta":28992,"kb_avail":562439976,"statfs":{"total":575970213888,"available":575938535424,"internally_reserved":0,"allocated":1941504,"data_stored":722634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29687808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"33","reported_epoch":"35","state":"active+clean","last_fresh":"2021-11-19T08:39:20.517528+0000","last_change":"2021-11-19T08:39:04.385737+0000","last_active":"2021-11-19T08:39:20.517528+0000","last_peered":"2021-11-19T08:39:20.517528+0000","last_clean":"2021-11-19T08:39:20.517528+0000","last_became_active":"2021-11-19T08:39:04.385272+0000","last_became_peered":"2021-11-19T08:39:04.385272+0000","last_unstale":"2021-11-19T08:39:20.517528+0000","last_undegraded":"2021-11-19T08:39:20.517528+0000","last_fullsized":"2021-11-19T08:39:20.517528+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-11-19T08:38:43.221904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-11-19T08:38:43.221904+0000","last_clean_scrub_stamp":"2021-11-19T08:38:43.221904+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":35,"seq":150323855372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5060,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740092,"statfs":{"total":95995035648,"available":95989854208,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986191,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5060,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740092,"statfs":{"total":95995035648,"available":95989854208,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5124,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93740028,"statfs":{"total":95995035648,"available":95989788672,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 19 08:39:49 2021","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.50600000000000001}]},{"osd":1,"last update":"Fri Nov 19 08:39:49 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Nov 19 08:39:49 2021","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.45500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411350,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5188,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739964,"statfs":{"total":95995035648,"available":95989723136,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 19 08:39:36 2021","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.435,"5min":0.435,"15min":0.435},"last":0.32200000000000001}]},{"osd":1,"last update":"Fri Nov 19 08:39:36 2021","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.40300000000000002}]},{"osd":3,"last update":"Fri Nov 19 08:39:48 2021","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":7.7690000000000001,"5min":7.7690000000000001,"15min":7.7690000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":7.6580000000000004,"5min":7.6580000000000004,"15min":7.6580000000000004},"last":0.55700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705693,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5252,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739900,"statfs":{"total":95995035648,"available":95989657600,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 19 08:39:15 2021","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.29799999999999999}]},{"osd":2,"last update":"Fri Nov 19 08:39:36 2021","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.315,"15min":0.315},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.317},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"last":0.34499999999999997}]},{"osd":3,"last update":"Fri Nov 19 08:39:52 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.58699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542169,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5252,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739900,"statfs":{"total":95995035648,"available":95989657600,"internally_reserved":0,"allocated":323584,"data_stored":120439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 19 08:39:17 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.44,"5min":0.44,"15min":0.44},"last":0.23999999999999999}]},{"osd":2,"last update":"Fri Nov 19 08:39:35 2021","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.36299999999999999}]},{"osd":3,"last update":"Fri Nov 19 08:39:50 2021","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-11-19T08:40:05.537 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2021-11-19T08:40:05.537 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-11-19T08:40:05.538 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2021-11-19T08:40:05.538 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- ceph health --format=json 2021-11-19T08:40:05.984 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/mon.smithi023/config 2021-11-19T08:40:06.769 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:06 smithi023 conmon[33490]: cluster 2021-11-19T08:40:04.821339+0000 mgr.smithi023.ehlpcl (mgr.14164) 170 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:06.769 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:06 smithi023 conmon[33490]: audit 2021-11-19T08:40:05.058588+0000 mgr.smithi023.ehlpcl (mgr.14164) 171 : audit [DBG] from='client.14398 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-19T08:40:07.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:06 smithi134 conmon[36363]: cluster 2021-11-19T08:40:04.821339+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:40:07.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:06 smithi134 conmon[36363]: ) 170 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:07.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:06 smithi134 conmon[36363]: audit 2021-11-19T08:40:05.058588+0000 mgr.smithi023.ehlpcl (mgr.14164) 171 : audit [DBG] from='client.14398 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-11-19T08:40:07.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.065+0000 7f61d4d2c700 1 -- 172.21.15.23:0/2922915722 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f7a80 msgr2=0x7f61d00f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:07.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.065+0000 7f61d4d2c700 1 --2- 172.21.15.23:0/2922915722 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f7a80 0x7f61d00f7ee0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f61c4004660 tx=0x7f61c4009b30).stop 2021-11-19T08:40:07.053 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.065+0000 7f61d4d2c700 1 -- 172.21.15.23:0/2922915722 shutdown_connections 2021-11-19T08:40:07.053 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.065+0000 7f61d4d2c700 1 --2- 172.21.15.23:0/2922915722 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f7a80 0x7f61d00f7ee0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:07.053 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.065+0000 7f61d4d2c700 1 --2- 172.21.15.23:0/2922915722 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61d00f67e0 0x7f61d00f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-19T08:40:07.053 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.065+0000 7f61d4d2c700 1 -- 172.21.15.23:0/2922915722 >> 172.21.15.23:0/2922915722 conn(0x7f61d00f1f20 msgr2=0x7f61d00f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:07.054 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.065+0000 7f61d4d2c700 1 -- 172.21.15.23:0/2922915722 shutdown_connections 2021-11-19T08:40:07.054 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.065+0000 7f61d4d2c700 1 -- 172.21.15.23:0/2922915722 wait complete. 2021-11-19T08:40:07.054 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.066+0000 7f61d4d2c700 1 Processor -- start 2021-11-19T08:40:07.054 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.066+0000 7f61d4d2c700 1 -- start start 2021-11-19T08:40:07.055 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.066+0000 7f61d4d2c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f67e0 0x7f61d01070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:07.055 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.066+0000 7f61d4d2c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61d00f7a80 0x7f61d01075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:07.055 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.066+0000 7f61d4d2c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d0107c00 con 0x7f61d00f67e0 2021-11-19T08:40:07.055 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.066+0000 7f61d4d2c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d01084b0 con 0x7f61d00f7a80 2021-11-19T08:40:07.055 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.066+0000 7f61ce59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f67e0 0x7f61d01070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:07.056 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.066+0000 7f61ce59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f67e0 0x7f61d01070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46842/0 (socket says 172.21.15.23:46842) 2021-11-19T08:40:07.056 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.066+0000 7f61ce59c700 1 -- 172.21.15.23:0/723930140 learned_addr learned my addr 172.21.15.23:0/723930140 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:07.056 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.066+0000 7f61cdd9b700 1 --2- 172.21.15.23:0/723930140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61d00f7a80 0x7f61d01075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:07.056 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.066+0000 7f61ce59c700 1 -- 172.21.15.23:0/723930140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61d00f7a80 msgr2=0x7f61d01075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:07.057 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.066+0000 7f61ce59c700 1 --2- 172.21.15.23:0/723930140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61d00f7a80 0x7f61d01075c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:07.057 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.066+0000 7f61ce59c700 1 -- 172.21.15.23:0/723930140 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c4005040 con 0x7f61d00f67e0 2021-11-19T08:40:07.057 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.066+0000 7f61ce59c700 1 --2- 172.21.15.23:0/723930140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f67e0 0x7f61d01070d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f61c00093f0 tx=0x7f61c0006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:07.057 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.067+0000 7f61beffd700 1 -- 172.21.15.23:0/723930140 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61c0009ea0 con 0x7f61d00f67e0 2021-11-19T08:40:07.058 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.067+0000 7f61beffd700 1 -- 172.21.15.23:0/723930140 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61c0013af0 con 0x7f61d00f67e0 2021-11-19T08:40:07.058 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.067+0000 7f61beffd700 1 -- 172.21.15.23:0/723930140 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61c000dad0 con 0x7f61d00f67e0 2021-11-19T08:40:07.058 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.067+0000 7f61d4d2c700 1 -- 172.21.15.23:0/723930140 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d0108790 con 0x7f61d00f67e0 2021-11-19T08:40:07.058 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.067+0000 7f61d4d2c700 1 -- 172.21.15.23:0/723930140 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d0108d50 con 0x7f61d00f67e0 2021-11-19T08:40:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.068+0000 7f61d4d2c700 1 -- 172.21.15.23:0/723930140 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d0059100 con 0x7f61d00f67e0 2021-11-19T08:40:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.068+0000 7f61beffd700 1 -- 172.21.15.23:0/723930140 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f61c001a060 con 0x7f61d00f67e0 2021-11-19T08:40:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.068+0000 7f61beffd700 1 --2- 172.21.15.23:0/723930140 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61b805c2a0 0x7f61b805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.069+0000 7f61beffd700 1 -- 172.21.15.23:0/723930140 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f61c0075b40 con 0x7f61d00f67e0 2021-11-19T08:40:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.069+0000 7f61cdd9b700 1 --2- 172.21.15.23:0/723930140 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61b805c2a0 0x7f61b805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.069+0000 7f61cdd9b700 1 --2- 172.21.15.23:0/723930140 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61b805c2a0 0x7f61b805e760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f61c4013580 tx=0x7f61c4009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.072+0000 7f61beffd700 1 -- 172.21.15.23:0/723930140 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61c0048b60 con 0x7f61d00f67e0 2021-11-19T08:40:07.280 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.295+0000 7f61d4d2c700 1 -- 172.21.15.23:0/723930140 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f61d00ff0b0 con 0x7f61d00f67e0 2021-11-19T08:40:07.280 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.295+0000 7f61beffd700 1 -- 172.21.15.23:0/723930140 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f61c004c180 con 0x7f61d00f67e0 2021-11-19T08:40:07.281 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:07.282 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.297+0000 7f61d4d2c700 1 -- 172.21.15.23:0/723930140 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61b805c2a0 msgr2=0x7f61b805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:07.282 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.297+0000 7f61d4d2c700 1 --2- 172.21.15.23:0/723930140 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61b805c2a0 0x7f61b805e760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f61c4013580 tx=0x7f61c4009b30).stop 2021-11-19T08:40:07.282 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.297+0000 7f61d4d2c700 1 -- 172.21.15.23:0/723930140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f67e0 msgr2=0x7f61d01070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:07.283 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.297+0000 7f61d4d2c700 1 --2- 172.21.15.23:0/723930140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f67e0 0x7f61d01070d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f61c00093f0 tx=0x7f61c0006ec0).stop 2021-11-19T08:40:07.283 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.297+0000 7f61d4d2c700 1 -- 172.21.15.23:0/723930140 shutdown_connections 2021-11-19T08:40:07.283 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.297+0000 7f61d4d2c700 1 --2- 172.21.15.23:0/723930140 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61b805c2a0 0x7f61b805e760 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:07.283 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.297+0000 7f61d4d2c700 1 --2- 172.21.15.23:0/723930140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d00f67e0 0x7f61d01070d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:07.284 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.297+0000 7f61d4d2c700 1 --2- 172.21.15.23:0/723930140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61d00f7a80 0x7f61d01075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:07.284 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.297+0000 7f61d4d2c700 1 -- 172.21.15.23:0/723930140 >> 172.21.15.23:0/723930140 conn(0x7f61d00f1f20 msgr2=0x7f61d00fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:07.284 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.297+0000 7f61d4d2c700 1 -- 172.21.15.23:0/723930140 shutdown_connections 2021-11-19T08:40:07.284 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:07.297+0000 7f61d4d2c700 1 -- 172.21.15.23:0/723930140 wait complete. 2021-11-19T08:40:07.693 INFO:teuthology.orchestra.run.smithi023.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-11-19T08:40:07.694 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2021-11-19T08:40:07.694 INFO:tasks.cephadm:Setup complete, yielding 2021-11-19T08:40:07.694 INFO:teuthology.run_tasks:Running task print... 2021-11-19T08:40:07.706 INFO:teuthology.task.print:**** done starting v16.2.4 2021-11-19T08:40:07.707 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-19T08:40:07.734 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi023.front.sepia.ceph.com 2021-11-19T08:40:07.735 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch status' 2021-11-19T08:40:08.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:07 smithi134 conmon[36363]: audit 2021-11-19T08:40:07.296401+0000 mon.smithi023 (mon.0) 446 : audit [DBG] 2021-11-19T08:40:08.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:07 smithi134 conmon[36363]: from='client.? 172.21.15.23:0/723930140' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-11-19T08:40:08.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:07 smithi023 conmon[33490]: audit 2021-11-19T 2021-11-19T08:40:08.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:07 smithi023 conmon[33490]: 08:40:07.296401+0000 mon.smithi023 (mon.0) 446 : audit [DBG] from='client.? 172.21.15.23:0/723930140' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-11-19T08:40:08.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:08 smithi023 conmon[33490]: cluster 2021-11-19T08:40:06.821786+0000 mgr.smithi023.ehlpcl (mgr.14164) 172 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:09.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:08 smithi134 conmon[36363]: cluster 2021-11-19T08:40:06.821786+0000 mgr.smithi023.ehlpcl (mgr.14164) 172 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:09.186 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.199+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/4168252127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b40f55c0 msgr2=0x7fa0b40f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:09.186 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.199+0000 7fa0b9cf6700 1 --2- 172.21.15.23:0/4168252127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b40f55c0 0x7fa0b40f59e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fa0a4004660 tx=0x7fa0a4009b30).stop 2021-11-19T08:40:09.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.199+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/4168252127 shutdown_connections 2021-11-19T08:40:09.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.199+0000 7fa0b9cf6700 1 --2- 172.21.15.23:0/4168252127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0b40f5ed0 0x7fa0b40f6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:09.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.199+0000 7fa0b9cf6700 1 --2- 172.21.15.23:0/4168252127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b40f55c0 0x7fa0b40f59e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:09.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.199+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/4168252127 >> 172.21.15.23:0/4168252127 conn(0x7fa0b40f0dc0 msgr2=0x7fa0b40f31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:09.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.199+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/4168252127 shutdown_connections 2021-11-19T08:40:09.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.199+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/4168252127 wait complete. 2021-11-19T08:40:09.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.199+0000 7fa0b9cf6700 1 Processor -- start 2021-11-19T08:40:09.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.199+0000 7fa0b9cf6700 1 -- start start 2021-11-19T08:40:09.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa0b9cf6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0b40f55c0 0x7fa0b4105e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:09.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa0b9cf6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b40f5ed0 0x7fa0b4106380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:09.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa0b9cf6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b4106930 con 0x7fa0b40f5ed0 2021-11-19T08:40:09.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa0b9cf6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b4106a70 con 0x7fa0b40f55c0 2021-11-19T08:40:09.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa0b2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b40f5ed0 0x7fa0b4106380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:09.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa0b2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b40f5ed0 0x7fa0b4106380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46854/0 (socket says 172.21.15.23:46854) 2021-11-19T08:40:09.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa0b2ffd700 1 -- 172.21.15.23:0/788394893 learned_addr learned my addr 172.21.15.23:0/788394893 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:09.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa0b37fe700 1 --2- 172.21.15.23:0/788394893 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0b40f55c0 0x7fa0b4105e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:09.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa0b2ffd700 1 -- 172.21.15.23:0/788394893 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0b40f55c0 msgr2=0x7fa0b4105e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:09.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa0b2ffd700 1 --2- 172.21.15.23:0/788394893 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0b40f55c0 0x7fa0b4105e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:09.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa0b2ffd700 1 -- 172.21.15.23:0/788394893 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a4005040 con 0x7fa0b40f5ed0 2021-11-19T08:40:09.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa0b2ffd700 1 --2- 172.21.15.23:0/788394893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b40f5ed0 0x7fa0b4106380 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fa0a800f940 tx=0x7fa0a8009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:09.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa09bfff700 1 -- 172.21.15.23:0/788394893 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0a8009500 con 0x7fa0b40f5ed0 2021-11-19T08:40:09.192 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/788394893 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b4107410 con 0x7fa0b40f5ed0 2021-11-19T08:40:09.192 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa09bfff700 1 -- 172.21.15.23:0/788394893 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0a8009660 con 0x7fa0b40f5ed0 2021-11-19T08:40:09.192 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa09bfff700 1 -- 172.21.15.23:0/788394893 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0a800de30 con 0x7fa0b40f5ed0 2021-11-19T08:40:09.192 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.200+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/788394893 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b41079d0 con 0x7fa0b40f5ed0 2021-11-19T08:40:09.193 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.201+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/788394893 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b4045ad0 con 0x7fa0b40f5ed0 2021-11-19T08:40:09.194 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.202+0000 7fa09bfff700 1 -- 172.21.15.23:0/788394893 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa0a800e040 con 0x7fa0b40f5ed0 2021-11-19T08:40:09.194 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.202+0000 7fa09bfff700 1 --2- 172.21.15.23:0/788394893 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa09c05c2b0 0x7fa09c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:09.194 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.202+0000 7fa09bfff700 1 -- 172.21.15.23:0/788394893 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fa0a8078630 con 0x7fa0b40f5ed0 2021-11-19T08:40:09.194 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.202+0000 7fa0b37fe700 1 --2- 172.21.15.23:0/788394893 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa09c05c2b0 0x7fa09c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:09.195 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.203+0000 7fa0b37fe700 1 --2- 172.21.15.23:0/788394893 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa09c05c2b0 0x7fa09c05e770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fa0a4009f50 tx=0x7fa0a4009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:09.195 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.206+0000 7fa09bfff700 1 -- 172.21.15.23:0/788394893 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0a804b570 con 0x7fa0b40f5ed0 2021-11-19T08:40:09.383 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.398+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/788394893 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0b405f7a0 con 0x7fa09c05c2b0 2021-11-19T08:40:09.384 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.400+0000 7fa09bfff700 1 -- 172.21.15.23:0/788394893 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fa0b405f7a0 con 0x7fa09c05c2b0 2021-11-19T08:40:09.385 INFO:teuthology.orchestra.run.smithi023.stdout:Backend: cephadm 2021-11-19T08:40:09.385 INFO:teuthology.orchestra.run.smithi023.stdout:Available: Yes 2021-11-19T08:40:09.385 INFO:teuthology.orchestra.run.smithi023.stdout:Paused: No 2021-11-19T08:40:09.386 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.401+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/788394893 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa09c05c2b0 msgr2=0x7fa09c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:09.387 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.401+0000 7fa0b9cf6700 1 --2- 172.21.15.23:0/788394893 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa09c05c2b0 0x7fa09c05e770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fa0a4009f50 tx=0x7fa0a4009b30).stop 2021-11-19T08:40:09.387 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.401+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/788394893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b40f5ed0 msgr2=0x7fa0b4106380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:09.387 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.401+0000 7fa0b9cf6700 1 --2- 172.21.15.23:0/788394893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b40f5ed0 0x7fa0b4106380 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fa0a800f940 tx=0x7fa0a8009de0).stop 2021-11-19T08:40:09.387 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.401+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/788394893 shutdown_connections 2021-11-19T08:40:09.388 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.401+0000 7fa0b9cf6700 1 --2- 172.21.15.23:0/788394893 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0b40f55c0 0x7fa0b4105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:09.388 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.401+0000 7fa0b9cf6700 1 --2- 172.21.15.23:0/788394893 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa09c05c2b0 0x7fa09c05e770 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:09.388 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.401+0000 7fa0b9cf6700 1 --2- 172.21.15.23:0/788394893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b40f5ed0 0x7fa0b4106380 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:09.388 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.401+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/788394893 >> 172.21.15.23:0/788394893 conn(0x7fa0b40f0dc0 msgr2=0x7fa0b40fa350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:09.389 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.401+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/788394893 shutdown_connections 2021-11-19T08:40:09.389 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:09.401+0000 7fa0b9cf6700 1 -- 172.21.15.23:0/788394893 wait complete. 2021-11-19T08:40:09.759 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch ps' 2021-11-19T08:40:10.848 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:10 smithi023 conmon[33490]: cluster 2021-11-19T08:40:08.822126+0000 mgr.smithi023.ehlpcl (mgr.14164) 173 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:10.848 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:10 smithi023 conmon[33490]: audit 2021-11-19T08:40:09.400555+0000 mgr.smithi023.ehlpcl (mgr.14164) 174 : audit [DBG] from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:40:11.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:10 smithi134 conmon[36363]: cluster 2021-11-19T08:40:08.822126+0000 mgr.smithi023.ehlpcl (mgr.14164) 173 : cluster 2021-11-19T08:40:11.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:10 smithi134 conmon[36363]: [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:11.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:10 smithi134 conmon[36363]: audit 2021-11-19T08:40:09.400555+0000 mgr.smithi023.ehlpcl (mgr.14164) 174 : audit [DBG] from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:40:11.092 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.107+0000 7f3602861700 1 -- 172.21.15.23:0/3897386610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc0f6870 msgr2=0x7f35fc0f6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:11.092 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.107+0000 7f3602861700 1 --2- 172.21.15.23:0/3897386610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc0f6870 0x7f35fc0f6cd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f35f0004660 tx=0x7f35f0009b30).stop 2021-11-19T08:40:11.093 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.107+0000 7f3602861700 1 -- 172.21.15.23:0/3897386610 shutdown_connections 2021-11-19T08:40:11.093 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.107+0000 7f3602861700 1 --2- 172.21.15.23:0/3897386610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc0f6870 0x7f35fc0f6cd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:11.093 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.107+0000 7f3602861700 1 --2- 172.21.15.23:0/3897386610 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35fc0f55d0 0x7f35fc0f59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:11.093 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.107+0000 7f3602861700 1 -- 172.21.15.23:0/3897386610 >> 172.21.15.23:0/3897386610 conn(0x7f35fc0f0d30 msgr2=0x7f35fc0f3150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:11.093 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.107+0000 7f3602861700 1 -- 172.21.15.23:0/3897386610 shutdown_connections 2021-11-19T08:40:11.096 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.108+0000 7f3602861700 1 -- 172.21.15.23:0/3897386610 wait complete. 2021-11-19T08:40:11.097 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.108+0000 7f3602861700 1 Processor -- start 2021-11-19T08:40:11.097 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.108+0000 7f3602861700 1 -- start start 2021-11-19T08:40:11.097 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f3602861700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35fc0f55d0 0x7f35fc105e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:11.097 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f3602861700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc0f6870 0x7f35fc106370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:11.097 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f3602861700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc106920 con 0x7f35fc0f6870 2021-11-19T08:40:11.098 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f3602861700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc106a60 con 0x7f35fc0f55d0 2021-11-19T08:40:11.098 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f35fb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc0f6870 0x7f35fc106370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:11.098 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f35fb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc0f6870 0x7f35fc106370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46864/0 (socket says 172.21.15.23:46864) 2021-11-19T08:40:11.099 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f35fb7fe700 1 -- 172.21.15.23:0/3170630242 learned_addr learned my addr 172.21.15.23:0/3170630242 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:11.099 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f35fbfff700 1 --2- 172.21.15.23:0/3170630242 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35fc0f55d0 0x7f35fc105e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:11.099 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f35fb7fe700 1 -- 172.21.15.23:0/3170630242 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35fc0f55d0 msgr2=0x7f35fc105e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:11.099 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f35fb7fe700 1 --2- 172.21.15.23:0/3170630242 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35fc0f55d0 0x7f35fc105e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:11.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f35fb7fe700 1 -- 172.21.15.23:0/3170630242 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f0005040 con 0x7f35fc0f6870 2021-11-19T08:40:11.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f35fb7fe700 1 --2- 172.21.15.23:0/3170630242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc0f6870 0x7f35fc106370 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f35f0009f50 tx=0x7f35f0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:11.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f35f8ff9700 1 -- 172.21.15.23:0/3170630242 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35f0018070 con 0x7f35fc0f6870 2021-11-19T08:40:11.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f35f8ff9700 1 -- 172.21.15.23:0/3170630242 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35f0013410 con 0x7f35fc0f6870 2021-11-19T08:40:11.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f35f8ff9700 1 -- 172.21.15.23:0/3170630242 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35f001c3c0 con 0x7f35fc0f6870 2021-11-19T08:40:11.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f3602861700 1 -- 172.21.15.23:0/3170630242 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35fc1073a0 con 0x7f35fc0f6870 2021-11-19T08:40:11.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.109+0000 7f3602861700 1 -- 172.21.15.23:0/3170630242 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35fc107900 con 0x7f35fc0f6870 2021-11-19T08:40:11.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.110+0000 7f3602861700 1 -- 172.21.15.23:0/3170630242 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35fc045ad0 con 0x7f35fc0f6870 2021-11-19T08:40:11.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.111+0000 7f35f8ff9700 1 -- 172.21.15.23:0/3170630242 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f35f001d420 con 0x7f35fc0f6870 2021-11-19T08:40:11.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.111+0000 7f35f8ff9700 1 --2- 172.21.15.23:0/3170630242 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f35e405c2b0 0x7f35e405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:11.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.111+0000 7f35f8ff9700 1 -- 172.21.15.23:0/3170630242 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f35f0077730 con 0x7f35fc0f6870 2021-11-19T08:40:11.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.111+0000 7f35fbfff700 1 --2- 172.21.15.23:0/3170630242 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f35e405c2b0 0x7f35e405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:11.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.112+0000 7f35fbfff700 1 --2- 172.21.15.23:0/3170630242 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f35e405c2b0 0x7f35e405e770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f35fc0f66d0 tx=0x7f35ec00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:11.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.115+0000 7f35f8ff9700 1 -- 172.21.15.23:0/3170630242 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35f004a7b0 con 0x7f35fc0f6870 2021-11-19T08:40:11.286 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.301+0000 7f3602861700 1 -- 172.21.15.23:0/3170630242 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f35fc107b70 con 0x7f35e405c2b0 2021-11-19T08:40:11.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.312+0000 7f35f8ff9700 1 -- 172.21.15.23:0/3170630242 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2160 (secure 0 0 0) 0x7f35fc107b70 con 0x7f35e405c2b0 2021-11-19T08:40:11.297 INFO:teuthology.orchestra.run.smithi023.stdout:NAME HOST PORTS STATUS REFRESHED AGE VERSION IMAGE ID CONTAINER ID 2021-11-19T08:40:11.297 INFO:teuthology.orchestra.run.smithi023.stdout:alertmanager.smithi023 smithi023 *:9093,9094 running (2m) 45s ago 3m 0.20.0 0881eb8f169f f05d45ec5f60 2021-11-19T08:40:11.298 INFO:teuthology.orchestra.run.smithi023.stdout:crash.smithi023 smithi023 running (3m) 45s ago 3m 16.2.4 8d91d370c2b8 deb32df2a615 2021-11-19T08:40:11.298 INFO:teuthology.orchestra.run.smithi023.stdout:crash.smithi134 smithi134 running (3m) 54s ago 3m 16.2.4 8d91d370c2b8 896c50bf9c73 2021-11-19T08:40:11.298 INFO:teuthology.orchestra.run.smithi023.stdout:grafana.smithi023 smithi023 *:3000 running (2m) 45s ago 3m 6.7.4 557c83e11646 11402d223b65 2021-11-19T08:40:11.298 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.smithi023.ehlpcl smithi023 *:9283 running (4m) 45s ago 4m 16.2.4 8d91d370c2b8 53ac1133d0d4 2021-11-19T08:40:11.299 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.smithi134.vitoqy smithi134 *:8443,9283 running (3m) 54s ago 3m 16.2.4 8d91d370c2b8 29c284e46c92 2021-11-19T08:40:11.299 INFO:teuthology.orchestra.run.smithi023.stdout:mon.smithi023 smithi023 running (4m) 45s ago 4m 16.2.4 8d91d370c2b8 04c7af95bd68 2021-11-19T08:40:11.299 INFO:teuthology.orchestra.run.smithi023.stdout:mon.smithi134 smithi134 running (2m) 54s ago 2m 16.2.4 8d91d370c2b8 48f4b0e16c71 2021-11-19T08:40:11.299 INFO:teuthology.orchestra.run.smithi023.stdout:node-exporter.smithi023 smithi023 *:9100 running (3m) 45s ago 3m 0.18.1 e5a616e4b9cf e478f858309e 2021-11-19T08:40:11.299 INFO:teuthology.orchestra.run.smithi023.stdout:node-exporter.smithi134 smithi134 *:9100 running (2m) 54s ago 2m 0.18.1 e5a616e4b9cf aeb85395549e 2021-11-19T08:40:11.300 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 smithi023 running (2m) 45s ago 2m 16.2.4 8d91d370c2b8 6a285b8e3d67 2021-11-19T08:40:11.300 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 smithi023 running (2m) 45s ago 2m 16.2.4 8d91d370c2b8 a9b980e5d923 2021-11-19T08:40:11.300 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 smithi023 running (105s) 45s ago 104s 16.2.4 8d91d370c2b8 12c1a5f33082 2021-11-19T08:40:11.300 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3 smithi134 running (89s) 54s ago 89s 16.2.4 8d91d370c2b8 82fa2db896f2 2021-11-19T08:40:11.301 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4 smithi134 running (73s) 54s ago 73s 16.2.4 8d91d370c2b8 13669f99ae66 2021-11-19T08:40:11.301 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5 smithi134 running (57s) 54s ago 56s 16.2.4 8d91d370c2b8 dd7ee7a28e97 2021-11-19T08:40:11.301 INFO:teuthology.orchestra.run.smithi023.stdout:prometheus.smithi023 smithi023 *:9095 running (2m) 45s ago 3m 2.18.1 de242295e225 91f34c02e915 2021-11-19T08:40:11.302 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.313+0000 7f3602861700 1 -- 172.21.15.23:0/3170630242 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f35e405c2b0 msgr2=0x7f35e405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:11.302 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.313+0000 7f3602861700 1 --2- 172.21.15.23:0/3170630242 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f35e405c2b0 0x7f35e405e770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f35fc0f66d0 tx=0x7f35ec00b040).stop 2021-11-19T08:40:11.303 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.313+0000 7f3602861700 1 -- 172.21.15.23:0/3170630242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc0f6870 msgr2=0x7f35fc106370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:11.303 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.313+0000 7f3602861700 1 --2- 172.21.15.23:0/3170630242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc0f6870 0x7f35fc106370 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f35f0009f50 tx=0x7f35f0009b30).stop 2021-11-19T08:40:11.303 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.313+0000 7f3602861700 1 -- 172.21.15.23:0/3170630242 shutdown_connections 2021-11-19T08:40:11.304 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.313+0000 7f3602861700 1 --2- 172.21.15.23:0/3170630242 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35fc0f55d0 0x7f35fc105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:11.304 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.313+0000 7f3602861700 1 --2- 172.21.15.23:0/3170630242 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f35e405c2b0 0x7f35e405e770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:11.304 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.313+0000 7f3602861700 1 --2- 172.21.15.23:0/3170630242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc0f6870 0x7f35fc106370 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:11.304 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.313+0000 7f3602861700 1 -- 172.21.15.23:0/3170630242 >> 172.21.15.23:0/3170630242 conn(0x7f35fc0f0d30 msgr2=0x7f35fc0fa2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:11.304 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.314+0000 7f3602861700 1 -- 172.21.15.23:0/3170630242 shutdown_connections 2021-11-19T08:40:11.305 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:11.314+0000 7f3602861700 1 -- 172.21.15.23:0/3170630242 wait complete. 2021-11-19T08:40:11.671 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch ls' 2021-11-19T08:40:12.870 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:12 smithi023 conmon[33490]: cluster 2021-11-19T08:40:10.822490+0000 mgr.smithi023.ehlpcl (mgr.14164) 175 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:12.871 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:12 smithi023 conmon[33490]: audit 2021-11-19T08:40:11.303373+0000 mgr.smithi023.ehlpcl (mgr.14164) 176 : audit [DBG] from='client.14410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:40:13.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:12 smithi134 conmon[36363]: cluster 2021-11-19T08:40:10.822490+0000 mgr.smithi023.ehlpcl (mgr.14164) 175 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:13.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:12 smithi134 conmon[36363]: audit 2021-11-19T08:40:11.303373+0000 mgr.smithi023.ehlpcl (mgr.14164) 176 : audit [DBG] from='client.14410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:40:13.110 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.123+0000 7f2c794f1700 1 -- 172.21.15.23:0/2727161492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f9b80 msgr2=0x7f2c740f9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:13.110 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.123+0000 7f2c794f1700 1 --2- 172.21.15.23:0/2727161492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f9b80 0x7f2c740f9fe0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f2c5c004660 tx=0x7f2c5c009b30).stop 2021-11-19T08:40:13.110 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.123+0000 7f2c794f1700 1 -- 172.21.15.23:0/2727161492 shutdown_connections 2021-11-19T08:40:13.110 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.123+0000 7f2c794f1700 1 --2- 172.21.15.23:0/2727161492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f9b80 0x7f2c740f9fe0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:13.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.123+0000 7f2c794f1700 1 --2- 172.21.15.23:0/2727161492 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c740f88e0 0x7f2c740f8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:13.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.123+0000 7f2c794f1700 1 -- 172.21.15.23:0/2727161492 >> 172.21.15.23:0/2727161492 conn(0x7f2c740f40e0 msgr2=0x7f2c740f64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:13.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.123+0000 7f2c794f1700 1 -- 172.21.15.23:0/2727161492 shutdown_connections 2021-11-19T08:40:13.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.123+0000 7f2c794f1700 1 -- 172.21.15.23:0/2727161492 wait complete. 2021-11-19T08:40:13.112 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.123+0000 7f2c794f1700 1 Processor -- start 2021-11-19T08:40:13.112 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.123+0000 7f2c794f1700 1 -- start start 2021-11-19T08:40:13.112 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c794f1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c740f88e0 0x7f2c74109180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:13.112 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c794f1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f9b80 0x7f2c74109670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:13.112 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c794f1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c74109c20 con 0x7f2c740f9b80 2021-11-19T08:40:13.113 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c794f1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c74109d60 con 0x7f2c740f88e0 2021-11-19T08:40:13.113 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c727fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f9b80 0x7f2c74109670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:13.113 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c727fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f9b80 0x7f2c74109670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46872/0 (socket says 172.21.15.23:46872) 2021-11-19T08:40:13.113 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c727fc700 1 -- 172.21.15.23:0/2710281186 learned_addr learned my addr 172.21.15.23:0/2710281186 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:13.114 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c72ffd700 1 --2- 172.21.15.23:0/2710281186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c740f88e0 0x7f2c74109180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:13.114 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c727fc700 1 -- 172.21.15.23:0/2710281186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c740f88e0 msgr2=0x7f2c74109180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:13.114 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c727fc700 1 --2- 172.21.15.23:0/2710281186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c740f88e0 0x7f2c74109180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:13.114 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c727fc700 1 -- 172.21.15.23:0/2710281186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c005040 con 0x7f2c740f9b80 2021-11-19T08:40:13.115 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c72ffd700 1 --2- 172.21.15.23:0/2710281186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c740f88e0 0x7f2c74109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:40:13.115 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c727fc700 1 --2- 172.21.15.23:0/2710281186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f9b80 0x7f2c74109670 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f2c5c007a40 tx=0x7f2c5c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:13.115 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c6b7fe700 1 -- 172.21.15.23:0/2710281186 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c5c018070 con 0x7f2c740f9b80 2021-11-19T08:40:13.115 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c794f1700 1 -- 172.21.15.23:0/2710281186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7410a6a0 con 0x7f2c740f9b80 2021-11-19T08:40:13.115 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c6b7fe700 1 -- 172.21.15.23:0/2710281186 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c5c013410 con 0x7f2c740f9b80 2021-11-19T08:40:13.116 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c6b7fe700 1 -- 172.21.15.23:0/2710281186 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c5c01c620 con 0x7f2c740f9b80 2021-11-19T08:40:13.116 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.124+0000 7f2c794f1700 1 -- 172.21.15.23:0/2710281186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7410ac00 con 0x7f2c740f9b80 2021-11-19T08:40:13.117 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.125+0000 7f2c794f1700 1 -- 172.21.15.23:0/2710281186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c74045ad0 con 0x7f2c740f9b80 2021-11-19T08:40:13.117 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.126+0000 7f2c6b7fe700 1 -- 172.21.15.23:0/2710281186 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2c5c01c8e0 con 0x7f2c740f9b80 2021-11-19T08:40:13.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.126+0000 7f2c6b7fe700 1 --2- 172.21.15.23:0/2710281186 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2c6005c2a0 0x7f2c6005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:13.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.127+0000 7f2c6b7fe700 1 -- 172.21.15.23:0/2710281186 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f2c5c077d20 con 0x7f2c740f9b80 2021-11-19T08:40:13.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.127+0000 7f2c72ffd700 1 --2- 172.21.15.23:0/2710281186 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2c6005c2a0 0x7f2c6005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:13.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.127+0000 7f2c72ffd700 1 --2- 172.21.15.23:0/2710281186 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2c6005c2a0 0x7f2c6005e760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f2c6c004fd0 tx=0x7f2c6c009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:13.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.130+0000 7f2c6b7fe700 1 -- 172.21.15.23:0/2710281186 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c5c04adb0 con 0x7f2c740f9b80 2021-11-19T08:40:13.302 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.315+0000 7f2c794f1700 1 -- 172.21.15.23:0/2710281186 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c7410ae20 con 0x7f2c6005c2a0 2021-11-19T08:40:13.305 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.320+0000 7f2c6b7fe700 1 -- 172.21.15.23:0/2710281186 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1116 (secure 0 0 0) 0x7f2c7410ae20 con 0x7f2c6005c2a0 2021-11-19T08:40:13.305 INFO:teuthology.orchestra.run.smithi023.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2021-11-19T08:40:13.306 INFO:teuthology.orchestra.run.smithi023.stdout:alertmanager 1/1 47s ago 4m count:1 2021-11-19T08:40:13.306 INFO:teuthology.orchestra.run.smithi023.stdout:crash 2/2 56s ago 4m * 2021-11-19T08:40:13.306 INFO:teuthology.orchestra.run.smithi023.stdout:grafana 1/1 47s ago 4m count:1 2021-11-19T08:40:13.306 INFO:teuthology.orchestra.run.smithi023.stdout:mgr 2/2 56s ago 4m count:2 2021-11-19T08:40:13.307 INFO:teuthology.orchestra.run.smithi023.stdout:mon 2/2 56s ago 3m smithi023:172.21.15.23=smithi023;smithi134:172.21.15.134=smithi134;count:2 2021-11-19T08:40:13.307 INFO:teuthology.orchestra.run.smithi023.stdout:node-exporter 2/2 56s ago 4m * 2021-11-19T08:40:13.307 INFO:teuthology.orchestra.run.smithi023.stdout:osd.unmanaged 6/6 56s ago - 2021-11-19T08:40:13.307 INFO:teuthology.orchestra.run.smithi023.stdout:prometheus 1/1 47s ago 4m count:1 2021-11-19T08:40:13.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.322+0000 7f2c794f1700 1 -- 172.21.15.23:0/2710281186 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2c6005c2a0 msgr2=0x7f2c6005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:13.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.322+0000 7f2c794f1700 1 --2- 172.21.15.23:0/2710281186 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2c6005c2a0 0x7f2c6005e760 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f2c6c004fd0 tx=0x7f2c6c009040).stop 2021-11-19T08:40:13.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.322+0000 7f2c794f1700 1 -- 172.21.15.23:0/2710281186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f9b80 msgr2=0x7f2c74109670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:13.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.322+0000 7f2c794f1700 1 --2- 172.21.15.23:0/2710281186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f9b80 0x7f2c74109670 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f2c5c007a40 tx=0x7f2c5c009b30).stop 2021-11-19T08:40:13.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.322+0000 7f2c794f1700 1 -- 172.21.15.23:0/2710281186 shutdown_connections 2021-11-19T08:40:13.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.322+0000 7f2c794f1700 1 --2- 172.21.15.23:0/2710281186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c740f88e0 0x7f2c74109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:13.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.322+0000 7f2c794f1700 1 --2- 172.21.15.23:0/2710281186 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2c6005c2a0 0x7f2c6005e760 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:13.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.322+0000 7f2c794f1700 1 --2- 172.21.15.23:0/2710281186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f9b80 0x7f2c74109670 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:13.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.322+0000 7f2c794f1700 1 -- 172.21.15.23:0/2710281186 >> 172.21.15.23:0/2710281186 conn(0x7f2c740f40e0 msgr2=0x7f2c740fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:13.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.322+0000 7f2c794f1700 1 -- 172.21.15.23:0/2710281186 shutdown_connections 2021-11-19T08:40:13.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:13.322+0000 7f2c794f1700 1 -- 172.21.15.23:0/2710281186 wait complete. 2021-11-19T08:40:13.593 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch host ls' 2021-11-19T08:40:14.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:14 smithi023 conmon[33490]: cluster 2021-11-19T08:40:12.822834+0000 mgr.smithi023.ehlpcl (mgr.14164) 177 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:14.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:14 smithi023 conmon[33490]: audit 2021-11-19T08:40:13.316989+0000 mgr.smithi023.ehlpcl (mgr.14164) 178 : audit [DBG] from='client.14414 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:40:15.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:14 smithi134 conmon[36363]: cluster 2021-11-19T08:40:12.822834+0000 mgr.smithi023.ehlpcl (mgr.14164) 177 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:15.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:14 smithi134 conmon[36363]: audit 2021-11-19T08:40:13.316989+0000 mgr.smithi023.ehlpcl (mgr.14164) 178 : audit [DBG] from='client.14414 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:40:15.062 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.076+0000 7f60102f7700 1 -- 172.21.15.23:0/2583854142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60080f55e0 msgr2=0x7f60080f5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.076+0000 7f60102f7700 1 --2- 172.21.15.23:0/2583854142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60080f55e0 0x7f60080f5a00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f60000099e0 tx=0x7f6000004d10).stop 2021-11-19T08:40:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.076+0000 7f60102f7700 1 -- 172.21.15.23:0/2583854142 shutdown_connections 2021-11-19T08:40:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.076+0000 7f60102f7700 1 --2- 172.21.15.23:0/2583854142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f60080f5ef0 0x7f60080f6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.076+0000 7f60102f7700 1 --2- 172.21.15.23:0/2583854142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60080f55e0 0x7f60080f5a00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.076+0000 7f60102f7700 1 -- 172.21.15.23:0/2583854142 >> 172.21.15.23:0/2583854142 conn(0x7f60080f0de0 msgr2=0x7f60080f3200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.076+0000 7f60102f7700 1 -- 172.21.15.23:0/2583854142 shutdown_connections 2021-11-19T08:40:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.076+0000 7f60102f7700 1 -- 172.21.15.23:0/2583854142 wait complete. 2021-11-19T08:40:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.076+0000 7f60102f7700 1 Processor -- start 2021-11-19T08:40:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.077+0000 7f60102f7700 1 -- start start 2021-11-19T08:40:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.077+0000 7f60102f7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f60080f55e0 0x7f6008109df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.077+0000 7f60102f7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60080f5ef0 0x7f600810c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.077+0000 7f60102f7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60080604e0 con 0x7f60080f5ef0 2021-11-19T08:40:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.077+0000 7f60102f7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6008060620 con 0x7f60080f55e0 2021-11-19T08:40:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.077+0000 7f600d892700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60080f5ef0 0x7f600810c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.077+0000 7f600d892700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60080f5ef0 0x7f600810c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46882/0 (socket says 172.21.15.23:46882) 2021-11-19T08:40:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.077+0000 7f600d892700 1 -- 172.21.15.23:0/341927918 learned_addr learned my addr 172.21.15.23:0/341927918 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.077+0000 7f600e093700 1 --2- 172.21.15.23:0/341927918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f60080f55e0 0x7f6008109df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:15.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.077+0000 7f600d892700 1 -- 172.21.15.23:0/341927918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f60080f55e0 msgr2=0x7f6008109df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:15.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.077+0000 7f600d892700 1 --2- 172.21.15.23:0/341927918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f60080f55e0 0x7f6008109df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:15.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.077+0000 7f600d892700 1 -- 172.21.15.23:0/341927918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6000009640 con 0x7f60080f5ef0 2021-11-19T08:40:15.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.077+0000 7f600d892700 1 --2- 172.21.15.23:0/341927918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60080f5ef0 0x7f600810c2f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f5ffc0026d0 tx=0x7f5ffc009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:15.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.077+0000 7f6006ffd700 1 -- 172.21.15.23:0/341927918 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ffc0079c0 con 0x7f60080f5ef0 2021-11-19T08:40:15.069 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.078+0000 7f60102f7700 1 -- 172.21.15.23:0/341927918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600810c8a0 con 0x7f60080f5ef0 2021-11-19T08:40:15.069 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.078+0000 7f6006ffd700 1 -- 172.21.15.23:0/341927918 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ffc00f040 con 0x7f60080f5ef0 2021-11-19T08:40:15.069 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.078+0000 7f6006ffd700 1 -- 172.21.15.23:0/341927918 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ffc0148b0 con 0x7f60080f5ef0 2021-11-19T08:40:15.070 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.078+0000 7f60102f7700 1 -- 172.21.15.23:0/341927918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600810cdb0 con 0x7f60080f5ef0 2021-11-19T08:40:15.070 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.079+0000 7f60102f7700 1 -- 172.21.15.23:0/341927918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60080f4a70 con 0x7f60080f5ef0 2021-11-19T08:40:15.070 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.079+0000 7f6006ffd700 1 -- 172.21.15.23:0/341927918 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5ffc007b20 con 0x7f60080f5ef0 2021-11-19T08:40:15.070 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.079+0000 7f6006ffd700 1 --2- 172.21.15.23:0/341927918 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5ff005c2a0 0x7f5ff005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:15.070 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.079+0000 7f6006ffd700 1 -- 172.21.15.23:0/341927918 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f5ffc075b00 con 0x7f60080f5ef0 2021-11-19T08:40:15.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.079+0000 7f600e093700 1 --2- 172.21.15.23:0/341927918 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5ff005c2a0 0x7f5ff005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:15.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.080+0000 7f600e093700 1 --2- 172.21.15.23:0/341927918 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5ff005c2a0 0x7f5ff005e760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f60000099b0 tx=0x7f6000004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:15.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.083+0000 7f6006ffd700 1 -- 172.21.15.23:0/341927918 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ffc048b00 con 0x7f60080f5ef0 2021-11-19T08:40:15.251 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.266+0000 7f60102f7700 1 -- 172.21.15.23:0/341927918 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f60080fdf20 con 0x7f5ff005c2a0 2021-11-19T08:40:15.253 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.268+0000 7f6006ffd700 1 -- 172.21.15.23:0/341927918 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+126 (secure 0 0 0) 0x7f60080fdf20 con 0x7f5ff005c2a0 2021-11-19T08:40:15.253 INFO:teuthology.orchestra.run.smithi023.stdout:HOST ADDR LABELS STATUS 2021-11-19T08:40:15.253 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 172.21.15.23 2021-11-19T08:40:15.254 INFO:teuthology.orchestra.run.smithi023.stdout:smithi134 smithi134 2021-11-19T08:40:15.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.270+0000 7f60102f7700 1 -- 172.21.15.23:0/341927918 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5ff005c2a0 msgr2=0x7f5ff005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:15.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.270+0000 7f60102f7700 1 --2- 172.21.15.23:0/341927918 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5ff005c2a0 0x7f5ff005e760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f60000099b0 tx=0x7f6000004d10).stop 2021-11-19T08:40:15.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.270+0000 7f60102f7700 1 -- 172.21.15.23:0/341927918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60080f5ef0 msgr2=0x7f600810c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:15.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.270+0000 7f60102f7700 1 --2- 172.21.15.23:0/341927918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60080f5ef0 0x7f600810c2f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f5ffc0026d0 tx=0x7f5ffc009de0).stop 2021-11-19T08:40:15.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.270+0000 7f60102f7700 1 -- 172.21.15.23:0/341927918 shutdown_connections 2021-11-19T08:40:15.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.270+0000 7f60102f7700 1 --2- 172.21.15.23:0/341927918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f60080f55e0 0x7f6008109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:15.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.270+0000 7f60102f7700 1 --2- 172.21.15.23:0/341927918 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5ff005c2a0 0x7f5ff005e760 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:15.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.270+0000 7f60102f7700 1 --2- 172.21.15.23:0/341927918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60080f5ef0 0x7f600810c2f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:15.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.270+0000 7f60102f7700 1 -- 172.21.15.23:0/341927918 >> 172.21.15.23:0/341927918 conn(0x7f60080f0de0 msgr2=0x7f60080fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:15.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.270+0000 7f60102f7700 1 -- 172.21.15.23:0/341927918 shutdown_connections 2021-11-19T08:40:15.258 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:15.270+0000 7f60102f7700 1 -- 172.21.15.23:0/341927918 wait complete. 2021-11-19T08:40:15.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- bash -c 'ceph orch device ls' 2021-11-19T08:40:16.908 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:16 smithi023 conmon[33490]: cluster 2021-11-19T08:40:14.823264+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:40:16.909 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:16 smithi023 conmon[33490]: .14164) 179 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:16.909 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:16 smithi023 conmon[33490]: audit 2021-11-19T08:40:15.268461+0000 mgr.smithi023.ehlpcl (mgr.14164) 180 : audit [DBG] from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:40:17.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:16 smithi134 conmon[36363]: cluster 2021-11-19T08:40:14.823264+0000 mgr.smithi023.ehlpcl (mgr.14164) 179 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:17.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:16 smithi134 conmon[36363]: audit 2021-11-19T08:40:15.268461+0000 mgr.smithi023.ehlpcl (mgr.14164) 180 : audit [DBG] from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:40:17.068 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.081+0000 7f5170672700 1 -- 172.21.15.23:0/2981281254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f516806a330 msgr2=0x7f516806a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:17.068 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.081+0000 7f5170672700 1 --2- 172.21.15.23:0/2981281254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f516806a330 0x7f516806a7b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f515c004660 tx=0x7f515c009b30).stop 2021-11-19T08:40:17.068 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.081+0000 7f5170672700 1 -- 172.21.15.23:0/2981281254 shutdown_connections 2021-11-19T08:40:17.069 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.081+0000 7f5170672700 1 --2- 172.21.15.23:0/2981281254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f516806a330 0x7f516806a7b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:17.069 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.081+0000 7f5170672700 1 --2- 172.21.15.23:0/2981281254 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f516806b940 0x7f5168069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:17.070 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.081+0000 7f5170672700 1 -- 172.21.15.23:0/2981281254 >> 172.21.15.23:0/2981281254 conn(0x7f51680f1e40 msgr2=0x7f51680f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:17.070 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.081+0000 7f5170672700 1 -- 172.21.15.23:0/2981281254 shutdown_connections 2021-11-19T08:40:17.070 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.081+0000 7f5170672700 1 -- 172.21.15.23:0/2981281254 wait complete. 2021-11-19T08:40:17.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.082+0000 7f5170672700 1 Processor -- start 2021-11-19T08:40:17.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.082+0000 7f5170672700 1 -- start start 2021-11-19T08:40:17.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.082+0000 7f5170672700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f516806a330 0x7f5168109150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:17.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.082+0000 7f5170672700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f516806b940 0x7f5168109640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:17.072 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.082+0000 7f5170672700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5168109bf0 con 0x7f516806a330 2021-11-19T08:40:17.072 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.082+0000 7f5170672700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5168109d30 con 0x7f516806b940 2021-11-19T08:40:17.072 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.082+0000 7f516e40e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f516806a330 0x7f5168109150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:17.072 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.082+0000 7f516e40e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f516806a330 0x7f5168109150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46888/0 (socket says 172.21.15.23:46888) 2021-11-19T08:40:17.073 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.082+0000 7f516e40e700 1 -- 172.21.15.23:0/2991439446 learned_addr learned my addr 172.21.15.23:0/2991439446 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:17.073 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.082+0000 7f516dc0d700 1 --2- 172.21.15.23:0/2991439446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f516806b940 0x7f5168109640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:17.073 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.082+0000 7f516e40e700 1 -- 172.21.15.23:0/2991439446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f516806b940 msgr2=0x7f5168109640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:17.073 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.082+0000 7f516e40e700 1 --2- 172.21.15.23:0/2991439446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f516806b940 0x7f5168109640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:17.073 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.082+0000 7f516e40e700 1 -- 172.21.15.23:0/2991439446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c005040 con 0x7f516806a330 2021-11-19T08:40:17.074 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.083+0000 7f516e40e700 1 --2- 172.21.15.23:0/2991439446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f516806a330 0x7f5168109150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f516000c7c0 tx=0x7f5160004d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:17.074 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.083+0000 7f5166ffd700 1 -- 172.21.15.23:0/2991439446 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f516000e680 con 0x7f516806a330 2021-11-19T08:40:17.074 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.083+0000 7f5166ffd700 1 -- 172.21.15.23:0/2991439446 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f516000ec90 con 0x7f516806a330 2021-11-19T08:40:17.074 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.083+0000 7f5170672700 1 -- 172.21.15.23:0/2991439446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f516810a6d0 con 0x7f516806a330 2021-11-19T08:40:17.075 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.083+0000 7f5166ffd700 1 -- 172.21.15.23:0/2991439446 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5160012d10 con 0x7f516806a330 2021-11-19T08:40:17.075 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.083+0000 7f5170672700 1 -- 172.21.15.23:0/2991439446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f516810ac90 con 0x7f516806a330 2021-11-19T08:40:17.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.084+0000 7f5170672700 1 -- 172.21.15.23:0/2991439446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5168045ad0 con 0x7f516806a330 2021-11-19T08:40:17.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.084+0000 7f5166ffd700 1 -- 172.21.15.23:0/2991439446 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5160019030 con 0x7f516806a330 2021-11-19T08:40:17.077 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.084+0000 7f5166ffd700 1 --2- 172.21.15.23:0/2991439446 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f515005c2b0 0x7f515005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:17.078 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.085+0000 7f5166ffd700 1 -- 172.21.15.23:0/2991439446 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f5160075bd0 con 0x7f516806a330 2021-11-19T08:40:17.078 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.085+0000 7f516dc0d700 1 --2- 172.21.15.23:0/2991439446 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f515005c2b0 0x7f515005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:17.078 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.085+0000 7f516dc0d700 1 --2- 172.21.15.23:0/2991439446 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f515005c2b0 0x7f515005e770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f515c004630 tx=0x7f515c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:17.078 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.088+0000 7f5166ffd700 1 -- 172.21.15.23:0/2991439446 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51600462b0 con 0x7f516806a330 2021-11-19T08:40:17.259 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.274+0000 7f5170672700 1 -- 172.21.15.23:0/2991439446 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5168101170 con 0x7f515005c2b0 2021-11-19T08:40:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.276+0000 7f5166ffd700 1 -- 172.21.15.23:0/2991439446 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+279 (secure 0 0 0) 0x7f5168101170 con 0x7f515005c2b0 2021-11-19T08:40:17.261 INFO:teuthology.orchestra.run.smithi023.stdout:Hostname Path Type Serial Size Health Ident Fault Available 2021-11-19T08:40:17.261 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 /dev/nvme0n1 ssd BTLE736103B14P0KGN 4000G Unknown N/A N/A No 2021-11-19T08:40:17.262 INFO:teuthology.orchestra.run.smithi023.stdout:smithi134 /dev/nvme0n1 ssd CVFT62420002400BGN 400G Unknown N/A N/A No 2021-11-19T08:40:17.263 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.277+0000 7f5170672700 1 -- 172.21.15.23:0/2991439446 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f515005c2b0 msgr2=0x7f515005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:17.263 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.277+0000 7f5170672700 1 --2- 172.21.15.23:0/2991439446 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f515005c2b0 0x7f515005e770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f515c004630 tx=0x7f515c009b30).stop 2021-11-19T08:40:17.263 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.277+0000 7f5170672700 1 -- 172.21.15.23:0/2991439446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f516806a330 msgr2=0x7f5168109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:17.263 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.277+0000 7f5170672700 1 --2- 172.21.15.23:0/2991439446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f516806a330 0x7f5168109150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f516000c7c0 tx=0x7f5160004d70).stop 2021-11-19T08:40:17.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.278+0000 7f5170672700 1 -- 172.21.15.23:0/2991439446 shutdown_connections 2021-11-19T08:40:17.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.278+0000 7f5170672700 1 --2- 172.21.15.23:0/2991439446 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f515005c2b0 0x7f515005e770 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:17.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.278+0000 7f5170672700 1 --2- 172.21.15.23:0/2991439446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f516806a330 0x7f5168109150 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:17.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.278+0000 7f5170672700 1 --2- 172.21.15.23:0/2991439446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f516806b940 0x7f5168109640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:17.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.278+0000 7f5170672700 1 -- 172.21.15.23:0/2991439446 >> 172.21.15.23:0/2991439446 conn(0x7f51680f1e40 msgr2=0x7f51680fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:17.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.278+0000 7f5170672700 1 -- 172.21.15.23:0/2991439446 shutdown_connections 2021-11-19T08:40:17.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:17.278+0000 7f5170672700 1 -- 172.21.15.23:0/2991439446 wait complete. 2021-11-19T08:40:17.640 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-19T08:40:17.654 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi023.front.sepia.ceph.com 2021-11-19T08:40:17.655 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs volume create cephfs --placement=4' 2021-11-19T08:40:18.789 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:18 smithi023 conmon[33490]: cluster 2021-11-19T08:40:16.823720+0000 mgr.smithi023.ehlpcl (mgr.14164) 181 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:18.790 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:18 smithi023 conmon[33490]: audit 2021-11-19T08:40:17.276202+0000 mgr.smithi023.ehlpcl (mgr.14164) 182 : audit [DBG] from='client.14422 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:40:19.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:18 smithi134 conmon[36363]: cluster 2021-11-19T08:40:16.823720+0000 mgr.smithi023.ehlpcl (mgr.14164) 181 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:19.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:18 smithi134 conmon[36363]: audit 2021-11-19T08:40:17.276202+0000 mgr.smithi023.ehlpcl (mgr.14164) 182 : audit [DBG] from='client.14422 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:40:19.043 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.053+0000 7f1540bf5700 1 -- 172.21.15.23:0/610975664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f153c0f6bb0 msgr2=0x7f153c0f6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:19.044 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.053+0000 7f1540bf5700 1 --2- 172.21.15.23:0/610975664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f153c0f6bb0 0x7f153c0f6fd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f152c004660 tx=0x7f152c009b30).stop 2021-11-19T08:40:19.044 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.053+0000 7f1540bf5700 1 -- 172.21.15.23:0/610975664 shutdown_connections 2021-11-19T08:40:19.044 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.054+0000 7f1540bf5700 1 --2- 172.21.15.23:0/610975664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f153c0f58b0 0x7f153c0f5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:19.044 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.054+0000 7f1540bf5700 1 --2- 172.21.15.23:0/610975664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f153c0f6bb0 0x7f153c0f6fd0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:19.045 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.054+0000 7f1540bf5700 1 -- 172.21.15.23:0/610975664 >> 172.21.15.23:0/610975664 conn(0x7f153c0f04c0 msgr2=0x7f153c0f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:19.045 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.054+0000 7f1540bf5700 1 -- 172.21.15.23:0/610975664 shutdown_connections 2021-11-19T08:40:19.045 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.054+0000 7f1540bf5700 1 -- 172.21.15.23:0/610975664 wait complete. 2021-11-19T08:40:19.045 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.054+0000 7f1540bf5700 1 Processor -- start 2021-11-19T08:40:19.046 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.054+0000 7f1540bf5700 1 -- start start 2021-11-19T08:40:19.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.054+0000 7f1540bf5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f153c0f58b0 0x7f153c106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:19.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.054+0000 7f1540bf5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f153c0f6bb0 0x7f153c1074a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:19.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.054+0000 7f1540bf5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c107a50 con 0x7f153c0f58b0 2021-11-19T08:40:19.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.054+0000 7f1540bf5700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c107b90 con 0x7f153c0f6bb0 2021-11-19T08:40:19.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.055+0000 7f1539d9b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f153c0f6bb0 0x7f153c1074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:19.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.055+0000 7f1539d9b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f153c0f6bb0 0x7f153c1074a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.23:51546/0 (socket says 172.21.15.23:51546) 2021-11-19T08:40:19.050 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.055+0000 7f1539d9b700 1 -- 172.21.15.23:0/4163613716 learned_addr learned my addr 172.21.15.23:0/4163613716 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:19.050 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.055+0000 7f153a59c700 1 --2- 172.21.15.23:0/4163613716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f153c0f58b0 0x7f153c106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:19.050 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.055+0000 7f153a59c700 1 -- 172.21.15.23:0/4163613716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f153c0f6bb0 msgr2=0x7f153c1074a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:19.051 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.055+0000 7f153a59c700 1 --2- 172.21.15.23:0/4163613716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f153c0f6bb0 0x7f153c1074a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:19.051 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.055+0000 7f153a59c700 1 -- 172.21.15.23:0/4163613716 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152c005040 con 0x7f153c0f58b0 2021-11-19T08:40:19.051 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.055+0000 7f1539d9b700 1 --2- 172.21.15.23:0/4163613716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f153c0f6bb0 0x7f153c1074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-11-19T08:40:19.051 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.055+0000 7f153a59c700 1 --2- 172.21.15.23:0/4163613716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f153c0f58b0 0x7f153c106fb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f152c009830 tx=0x7f152c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:19.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.055+0000 7f152affd700 1 -- 172.21.15.23:0/4163613716 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f152c018070 con 0x7f153c0f58b0 2021-11-19T08:40:19.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.055+0000 7f152affd700 1 -- 172.21.15.23:0/4163613716 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f152c013410 con 0x7f153c0f58b0 2021-11-19T08:40:19.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.055+0000 7f1540bf5700 1 -- 172.21.15.23:0/4163613716 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153c1084d0 con 0x7f153c0f58b0 2021-11-19T08:40:19.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.055+0000 7f152affd700 1 -- 172.21.15.23:0/4163613716 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f152c01de30 con 0x7f153c0f58b0 2021-11-19T08:40:19.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.055+0000 7f1540bf5700 1 -- 172.21.15.23:0/4163613716 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c108a00 con 0x7f153c0f58b0 2021-11-19T08:40:19.053 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.056+0000 7f1540bf5700 1 -- 172.21.15.23:0/4163613716 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c0f4150 con 0x7f153c0f58b0 2021-11-19T08:40:19.053 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.057+0000 7f152affd700 1 -- 172.21.15.23:0/4163613716 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f152c01d420 con 0x7f153c0f58b0 2021-11-19T08:40:19.053 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.057+0000 7f152affd700 1 --2- 172.21.15.23:0/4163613716 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f152405c2b0 0x7f152405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:19.053 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.057+0000 7f152affd700 1 -- 172.21.15.23:0/4163613716 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f152c077610 con 0x7f153c0f58b0 2021-11-19T08:40:19.054 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.057+0000 7f1539d9b700 1 --2- 172.21.15.23:0/4163613716 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f152405c2b0 0x7f152405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:19.054 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.058+0000 7f1539d9b700 1 --2- 172.21.15.23:0/4163613716 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f152405c2b0 0x7f152405e770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f15300102e0 tx=0x7f153000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:19.054 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.061+0000 7f152affd700 1 -- 172.21.15.23:0/4163613716 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f152c04a690 con 0x7f153c0f58b0 2021-11-19T08:40:19.237 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:19.252+0000 7f1540bf5700 1 -- 172.21.15.23:0/4163613716 --> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f153c0fefc0 con 0x7f152405c2b0 2021-11-19T08:40:20.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:20 smithi134 conmon[36363]: cluster 2021-11-19T08:40:18.824087+0000 mgr.smithi023.ehlpcl (mgr.14164) 183 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:20.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:20 smithi134 conmon[36363]: audit 2021-11-19T08:40:19.254043+0000 mgr.smithi023.ehlpcl (mgr.14164) 184 : audit [DBG] from='client.14426 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:40:20.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:20 smithi134 conmon[36363]: audit 2021-11-19T08:40:19.254509+0000 mon.smithi023 (mon.0) 447 2021-11-19T08:40:20.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:20 smithi134 conmon[36363]: : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2021-11-19T08:40:20.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:20 smithi023 conmon[33490]: cluster 2021-11-19T08:40 2021-11-19T08:40:20.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:20 smithi023 conmon[33490]: :18.824087+0000 mgr.smithi023.ehlpcl (mgr.14164) 183 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:20.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:20 smithi023 conmon[33490]: audit 2021-11-19T08:40:19.254043+0000 mgr.smithi023.ehlpcl (mgr.14164) 184 : audit [DBG] from='client.14426 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2021-11-19T08:40:20.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:20 smithi023 conmon[33490]: audit 2021-11-19T08:40:19.254509+0000 mon.smithi023 (mon.0) 447 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2021-11-19T08:40:21.212 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:21.223+0000 7f152affd700 1 -- 172.21.15.23:0/4163613716 <== mgr.14164 v2:172.21.15.23:6800/3049298848 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f153c0fefc0 con 0x7f152405c2b0 2021-11-19T08:40:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:21.226+0000 7f1540bf5700 1 -- 172.21.15.23:0/4163613716 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f152405c2b0 msgr2=0x7f152405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:21.226+0000 7f1540bf5700 1 --2- 172.21.15.23:0/4163613716 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f152405c2b0 0x7f152405e770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f15300102e0 tx=0x7f153000b040).stop 2021-11-19T08:40:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:21.227+0000 7f1540bf5700 1 -- 172.21.15.23:0/4163613716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f153c0f58b0 msgr2=0x7f153c106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:21.227+0000 7f1540bf5700 1 --2- 172.21.15.23:0/4163613716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f153c0f58b0 0x7f153c106fb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f152c009830 tx=0x7f152c009b30).stop 2021-11-19T08:40:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:21.227+0000 7f1540bf5700 1 -- 172.21.15.23:0/4163613716 shutdown_connections 2021-11-19T08:40:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:21.227+0000 7f1540bf5700 1 --2- 172.21.15.23:0/4163613716 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f152405c2b0 0x7f152405e770 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:21.227+0000 7f1540bf5700 1 --2- 172.21.15.23:0/4163613716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f153c0f58b0 0x7f153c106fb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:21.227+0000 7f1540bf5700 1 --2- 172.21.15.23:0/4163613716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f153c0f6bb0 0x7f153c1074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:21.227+0000 7f1540bf5700 1 -- 172.21.15.23:0/4163613716 >> 172.21.15.23:0/4163613716 conn(0x7f153c0f04c0 msgr2=0x7f153c0fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:21.227+0000 7f1540bf5700 1 -- 172.21.15.23:0/4163613716 shutdown_connections 2021-11-19T08:40:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:21.227+0000 7f1540bf5700 1 -- 172.21.15.23:0/4163613716 wait complete. 2021-11-19T08:40:21.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:21 smithi134 conmon[36363]: audit 2021-11-19T08:40:20.180152+0000 mon.smithi023 (mon.0 2021-11-19T08:40:21.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:21 smithi134 conmon[36363]: ) 448 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2021-11-19T08:40:21.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:21 smithi134 conmon[36363]: cluster 2021-11-19T08:40:20.180229+0000 mon.smithi023 (mon.0) 449 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-11-19T08:40:21.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:21 smithi134 conmon[36363]: audit 2021-11-19T08:40:20.182252+0000 mon.smithi023 ( 2021-11-19T08:40:21.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:21 smithi134 conmon[36363]: mon.0) 450 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2021-11-19T08:40:21.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:21 smithi134 conmon[36363]: audit 2021-11-19 2021-11-19T08:40:21.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:21 smithi134 conmon[36363]: T08:40:20.914530+0000 mon.smithi023 (mon. 2021-11-19T08:40:21.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:21 smithi134 conmon[36363]: 0) 451 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:40:21.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:21 smithi134 conmon[36363]: audit 2021-11-19T 2021-11-19T08:40:21.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:21 smithi134 conmon[36363]: 08:40:20.914905+0000 mon.smithi023 (mon.0) 452 2021-11-19T08:40:21.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:21 smithi134 conmon[36363]: : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:40:21.567 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: audit 2021-11-19T08:40:20 2021-11-19T08:40:21.568 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: .180152+0000 mon.smithi023 (mon.0) 448 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2021-11-19T08:40:21.568 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: cluster 2021-11 2021-11-19T08:40:21.568 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: -19T08:40:20.180229+0000 mon.smithi023 (mon.0) 449 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2021-11-19T08:40:21.569 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: audit 2021 2021-11-19T08:40:21.569 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: -11-19T08:40:20.182252+0000 2021-11-19T08:40:21.570 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: mon.smithi023 (mon.0) 450 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2021-11-19T08:40:21.570 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: 2021-11-19T08:40:21.571 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:40:21.571 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: T08:40:20.914530+0000 mon.smithi023 (mon.0) 451 : audit 2021-11-19T08:40:21.571 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:40:21.572 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:40:21.572 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: T08:40:20.914905+0000 mon.smithi023 (mon.0) 452 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:40:21.573 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: 2021-11-19T08:40:21.573 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:21 smithi023 conmon[33490]: debug 2021-11-19T08:40:21.200+0000 7ff52e01e700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-19T08:40:21.575 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs dump' 2021-11-19T08:40:22.295 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: cluster 2021-11-19T08:40:20.824480+0000 mgr.smithi023.ehlpcl (mgr.14164) 185 : cluster [DBG] pgmap v147: 33 pgs: 32 unknown, 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:22.296 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: audit 2021-11-19T08:40:21.190354+0000 mon.smithi023 (mon.0) 453 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2021-11-19T08:40:22.296 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: cluster 2021-11-19T08:40:21.190577+0000 mon.smithi023 (mon.0) 454 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-11-19T08:40:22.297 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: audit 2021-11-19T08:40:21.200798+0000 mon.smithi023 (mon.0) 455 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2021-11-19T08:40:22.297 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: cluster 2021-11-19T08:40:21.201717+0000 mon.smithi023 (mon.0) 456 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-19T08:40:22.297 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: cluster 2021-11-19T08:40:21.201745+0000 mon.smithi023 (mon.0 2021-11-19T08:40:22.298 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: ) 457 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2021-11-19T08:40:22.298 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: cluster 2021-11-19T08:40:21.210495+0000 mon.smithi023 (mon.0) 458 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2021-11-19T08:40:22.298 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: audit 2021-11-19T08:40:21.210747+0000 mon.smithi023 (mon.0) 459 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2021-11-19T08:40:22.299 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: cluster 2021-11-19T08:40:21.210843+0000 mon.smithi023 (mon.0) 460 : cluster [DBG] fsmap cephfs:0 2021-11-19T08:40:22.299 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: cephadm 2021-11-19T08:40:21.215494+0000 mgr.smithi023.ehlpcl (mgr.14164) 186 : cephadm [INF] Saving service mds.cephfs spec with placement count:4 2021-11-19T08:40:22.299 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: audit 2021-11-19T08:40:21.223860+0000 mon.smithi023 (mon.0) 461 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:22.301 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: audit 2021-11-19T08:40:21.225175+0000 mon.smithi023 (mon.0) 462 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:40:22.302 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: audit 2021-11-19T08:40:21.722140+0000 mon.smithi023 (mon.0) 463 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:22.302 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: audit 2021-11-19T08:40:21.732898+0000 mon.smithi023 (mon.0) 464 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:22.302 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: audit 2021-11-19T08:40:21.733718+0000 mon.smithi023 (mon.0) 465 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi023.fuvzkr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-19T08:40:22.302 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: audit 2021-11-19T08:40 2021-11-19T08:40:22.303 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: :21.737475+0000 mon.smithi023 (mon.0) 466 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi023.fuvzkr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-19T08:40:22.303 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:40:22.303 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: T08:40:21.738229+0000 mon.smithi023 (mon.0) 467 : audit [DBG] 2021-11-19T08:40:22.303 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:40:22.303 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: cephadm 2021-11-19T08:40 2021-11-19T08:40:22.304 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: :21.739081+0000 mgr.smithi023.ehlpcl (mgr.14164) 187 : 2021-11-19T08:40:22.304 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:22 smithi023 conmon[33490]: cephadm [INF] Deploying daemon mds.cephfs.smithi023.fuvzkr on smithi023 2021-11-19T08:40:22.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: cluster 2021-11-19T08:40:20.824480+0000 mgr.smithi023.ehlpcl (mgr.14164) 185 2021-11-19T08:40:22.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: : cluster [DBG] pgmap v147: 33 pgs: 32 unknown, 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:22.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: audit 2021-11-19T08:40:21.190354+0000 mon.smithi023 (mon.0) 453 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2021-11-19T08:40:22.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: cluster 2021-11-19T08:40:21.190577+0000 mon.smithi023 (mon.0) 454 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-11-19T08:40:22.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: audit 2021-11-19T08:40:21.200798+0000 mon.smithi023 (mon.0) 455 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2021-11-19T08:40:22.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: cluster 2021-11-19T08:40:21.201717+0000 mon.smithi023 (mon.0) 456 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-11-19T08:40:22.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: cluster 2021-11-19T08:40:21.201745+0000 2021-11-19T08:40:22.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: mon.smithi023 (mon.0) 457 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2021-11-19T08:40:22.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: cluster 2021-11-19T08:40:21.210495+0000 mon.smithi023 (mon.0) 458 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2021-11-19T08:40:22.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: 2021-11-19T08:40:22.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: audit 2021-11 2021-11-19T08:40:22.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: -19T08:40:21.210747+0000 mon.smithi023 (mon.0) 459 : 2021-11-19T08:40:22.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2021-11-19T08:40:22.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: cluster 2021 2021-11-19T08:40:22.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: -11-19T08:40:21.210843+0000 mon.smithi023 2021-11-19T08:40:22.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: (mon.0) 460 : cluster [DBG] fsmap cephfs:0 2021-11-19T08:40:22.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: cephadm 2021-11-19T08 2021-11-19T08:40:22.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: :40:21.215494+0000 mgr.smithi023.ehlpcl (mgr.14164) 186 : cephadm 2021-11-19T08:40:22.545 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: [INF] Saving service mds.cephfs spec with placement count:4 2021-11-19T08:40:22.546 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: audit 2021-11-19T08 2021-11-19T08:40:22.546 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: :40:21.223860+0000 mon.smithi023 (mon.0 2021-11-19T08:40:22.546 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: ) 461 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:22.546 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:40:22.547 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: 11-19T08:40:21.225175+0000 mon.smithi023 ( 2021-11-19T08:40:22.547 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: mon.0) 462 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:40:22.547 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: audit 2021-11- 2021-11-19T08:40:22.548 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: 19T08:40:21.722140+0000 mon.smithi023 (mon.0) 463 : audit 2021-11-19T08:40:22.548 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:22.548 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: audit 2021-11- 2021-11-19T08:40:22.548 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: 19T08:40:21.732898+0000 mon.smithi023 (mon 2021-11-19T08:40:22.548 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: .0) 464 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:22.549 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: audit 2021 2021-11-19T08:40:22.549 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: -11-19T08:40:21.733718+0000 2021-11-19T08:40:22.549 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: mon.smithi023 (mon.0) 465 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi023.fuvzkr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-19T08:40:22.549 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: 2021-11-19T08:40:22.550 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: audit 2021-11-19T08:40:21. 2021-11-19T08:40:22.550 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: 737475+0000 mon.smithi023 (mon.0) 466 : audit 2021-11-19T08:40:22.550 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi023.fuvzkr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-19T08:40:22.550 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: audit 2021-11-19T08: 2021-11-19T08:40:22.550 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: 40:21.738229+0000 mon.smithi023 (mon.0) 467 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:40:22.551 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: 2021-11-19T08:40:22.551 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: cephadm 2021-11-19T08:40 2021-11-19T08:40:22.551 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: :21.739081+0000 mgr.smithi023.ehlpcl (mgr.14164) 187 : 2021-11-19T08:40:22.551 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:22 smithi134 conmon[36363]: cephadm [INF] Deploying daemon mds.cephfs.smithi023.fuvzkr on smithi023 2021-11-19T08:40:23.531 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:23 smithi023 conmon[33490]: cluster 2021-11-19T08:40:22.213912+0000 mon.smithi023 (mon.0) 468 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2021-11-19T08:40:23.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:23 smithi134 conmon[36363]: cluster 2021-11-19T08:40:22.213912+0000 mon.smithi023 (mon.0) 468 : cluster [DBG] 2021-11-19T08:40:23.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:23 smithi134 conmon[36363]: osdmap e40: 6 total, 6 up, 6 in 2021-11-19T08:40:23.577 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.589+0000 7f06fd0b8700 1 -- 172.21.15.23:0/1607252903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f80f55c0 msgr2=0x7f06f80f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:23.577 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.589+0000 7f06fd0b8700 1 --2- 172.21.15.23:0/1607252903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f80f55c0 0x7f06f80f59e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f06f0009a30 tx=0x7f06f0004c50).stop 2021-11-19T08:40:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.589+0000 7f06fd0b8700 1 -- 172.21.15.23:0/1607252903 shutdown_connections 2021-11-19T08:40:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.589+0000 7f06fd0b8700 1 --2- 172.21.15.23:0/1607252903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06f80f6860 0x7f06f80f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.589+0000 7f06fd0b8700 1 --2- 172.21.15.23:0/1607252903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f80f55c0 0x7f06f80f59e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:23.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.589+0000 7f06fd0b8700 1 -- 172.21.15.23:0/1607252903 >> 172.21.15.23:0/1607252903 conn(0x7f06f80f0d40 msgr2=0x7f06f80f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:23.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.589+0000 7f06fd0b8700 1 -- 172.21.15.23:0/1607252903 shutdown_connections 2021-11-19T08:40:23.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.589+0000 7f06fd0b8700 1 -- 172.21.15.23:0/1607252903 wait complete. 2021-11-19T08:40:23.580 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06fd0b8700 1 Processor -- start 2021-11-19T08:40:23.581 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06fd0b8700 1 -- start start 2021-11-19T08:40:23.581 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06fd0b8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06f80f55c0 0x7f06f8109cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:23.581 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06fd0b8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f80f6860 0x7f06f810c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:23.581 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06fd0b8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f8060500 con 0x7f06f80f6860 2021-11-19T08:40:23.582 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06fd0b8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f8060640 con 0x7f06f80f55c0 2021-11-19T08:40:23.582 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06f659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f80f6860 0x7f06f810c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:23.582 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06f659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f80f6860 0x7f06f810c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46908/0 (socket says 172.21.15.23:46908) 2021-11-19T08:40:23.582 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06f659c700 1 -- 172.21.15.23:0/2911338162 learned_addr learned my addr 172.21.15.23:0/2911338162 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:23.582 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06f6d9d700 1 --2- 172.21.15.23:0/2911338162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06f80f55c0 0x7f06f8109cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:23.583 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06f659c700 1 -- 172.21.15.23:0/2911338162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06f80f55c0 msgr2=0x7f06f8109cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:23.583 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06f659c700 1 --2- 172.21.15.23:0/2911338162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06f80f55c0 0x7f06f8109cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:23.583 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06f659c700 1 -- 172.21.15.23:0/2911338162 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f0009640 con 0x7f06f80f6860 2021-11-19T08:40:23.583 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06f6d9d700 1 --2- 172.21.15.23:0/2911338162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06f80f55c0 0x7f06f8109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:40:23.584 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.590+0000 7f06f659c700 1 --2- 172.21.15.23:0/2911338162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f80f6860 0x7f06f810c1c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f06e000e7c0 tx=0x7f06e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:23.584 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.591+0000 7f06ef7fe700 1 -- 172.21.15.23:0/2911338162 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06e00092e0 con 0x7f06f80f6860 2021-11-19T08:40:23.584 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.591+0000 7f06ef7fe700 1 -- 172.21.15.23:0/2911338162 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06e000f040 con 0x7f06f80f6860 2021-11-19T08:40:23.584 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.591+0000 7f06fd0b8700 1 -- 172.21.15.23:0/2911338162 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f810c770 con 0x7f06f80f6860 2021-11-19T08:40:23.585 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.591+0000 7f06ef7fe700 1 -- 172.21.15.23:0/2911338162 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06e0014730 con 0x7f06f80f6860 2021-11-19T08:40:23.585 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.591+0000 7f06fd0b8700 1 -- 172.21.15.23:0/2911338162 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f810cd30 con 0x7f06f80f6860 2021-11-19T08:40:23.585 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.592+0000 7f06fd0b8700 1 -- 172.21.15.23:0/2911338162 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f8045ad0 con 0x7f06f80f6860 2021-11-19T08:40:23.586 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.592+0000 7f06ef7fe700 1 -- 172.21.15.23:0/2911338162 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f06e001b030 con 0x7f06f80f6860 2021-11-19T08:40:23.586 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.593+0000 7f06ef7fe700 1 --2- 172.21.15.23:0/2911338162 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f06e405c2a0 0x7f06e405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:23.587 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.593+0000 7f06ef7fe700 1 -- 172.21.15.23:0/2911338162 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f06e0077d80 con 0x7f06f80f6860 2021-11-19T08:40:23.587 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.593+0000 7f06f6d9d700 1 --2- 172.21.15.23:0/2911338162 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f06e405c2a0 0x7f06e405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:23.587 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.593+0000 7f06f6d9d700 1 --2- 172.21.15.23:0/2911338162 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f06e405c2a0 0x7f06e405e760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f06f0003380 tx=0x7f06f0004c50).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:23.587 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.597+0000 7f06ef7fe700 1 -- 172.21.15.23:0/2911338162 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06e007b060 con 0x7f06f80f6860 2021-11-19T08:40:23.800 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.814+0000 7f06fd0b8700 1 -- 172.21.15.23:0/2911338162 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f06f80586e0 con 0x7f06f80f6860 2021-11-19T08:40:23.801 INFO:teuthology.orchestra.run.smithi023.stdout:e2 2021-11-19T08:40:23.802 INFO:teuthology.orchestra.run.smithi023.stdout:enable_multiple, ever_enabled_multiple: 1,1 2021-11-19T08:40:23.802 INFO:teuthology.orchestra.run.smithi023.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-19T08:40:23.802 INFO:teuthology.orchestra.run.smithi023.stdout:legacy client fscid: 1 2021-11-19T08:40:23.802 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:23.803 INFO:teuthology.orchestra.run.smithi023.stdout:Filesystem 'cephfs' (1) 2021-11-19T08:40:23.803 INFO:teuthology.orchestra.run.smithi023.stdout:fs_name cephfs 2021-11-19T08:40:23.804 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 2 2021-11-19T08:40:23.804 INFO:teuthology.orchestra.run.smithi023.stdout:flags 12 2021-11-19T08:40:23.804 INFO:teuthology.orchestra.run.smithi023.stdout:created 2021-11-19T08:40:21.201543+0000 2021-11-19T08:40:23.805 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2021-11-19T08:40:21.201611+0000 2021-11-19T08:40:23.805 INFO:teuthology.orchestra.run.smithi023.stdout:tableserver 0 2021-11-19T08:40:23.805 INFO:teuthology.orchestra.run.smithi023.stdout:root 0 2021-11-19T08:40:23.806 INFO:teuthology.orchestra.run.smithi023.stdout:session_timeout 60 2021-11-19T08:40:23.806 INFO:teuthology.orchestra.run.smithi023.stdout:session_autoclose 300 2021-11-19T08:40:23.806 INFO:teuthology.orchestra.run.smithi023.stdout:max_file_size 1099511627776 2021-11-19T08:40:23.806 INFO:teuthology.orchestra.run.smithi023.stdout:required_client_features {} 2021-11-19T08:40:23.806 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure 0 2021-11-19T08:40:23.807 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure_osd_epoch 0 2021-11-19T08:40:23.807 INFO:teuthology.orchestra.run.smithi023.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-19T08:40:23.807 INFO:teuthology.orchestra.run.smithi023.stdout:max_mds 1 2021-11-19T08:40:23.807 INFO:teuthology.orchestra.run.smithi023.stdout:in 2021-11-19T08:40:23.808 INFO:teuthology.orchestra.run.smithi023.stdout:up {} 2021-11-19T08:40:23.808 INFO:teuthology.orchestra.run.smithi023.stdout:failed 2021-11-19T08:40:23.808 INFO:teuthology.orchestra.run.smithi023.stdout:damaged 2021-11-19T08:40:23.808 INFO:teuthology.orchestra.run.smithi023.stdout:stopped 2021-11-19T08:40:23.808 INFO:teuthology.orchestra.run.smithi023.stdout:data_pools [3] 2021-11-19T08:40:23.809 INFO:teuthology.orchestra.run.smithi023.stdout:metadata_pool 2 2021-11-19T08:40:23.809 INFO:teuthology.orchestra.run.smithi023.stdout:inline_data disabled 2021-11-19T08:40:23.809 INFO:teuthology.orchestra.run.smithi023.stdout:balancer 2021-11-19T08:40:23.809 INFO:teuthology.orchestra.run.smithi023.stdout:standby_count_wanted 0 2021-11-19T08:40:23.810 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:23.810 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:23.811 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.815+0000 7f06ef7fe700 1 -- 172.21.15.23:0/2911338162 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1003 (secure 0 0 0) 0x7f06e0019020 con 0x7f06f80f6860 2021-11-19T08:40:23.811 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.817+0000 7f06fd0b8700 1 -- 172.21.15.23:0/2911338162 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f06e405c2a0 msgr2=0x7f06e405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:23.812 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.817+0000 7f06fd0b8700 1 --2- 172.21.15.23:0/2911338162 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f06e405c2a0 0x7f06e405e760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f06f0003380 tx=0x7f06f0004c50).stop 2021-11-19T08:40:23.812 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.817+0000 7f06fd0b8700 1 -- 172.21.15.23:0/2911338162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f80f6860 msgr2=0x7f06f810c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:23.812 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.817+0000 7f06fd0b8700 1 --2- 172.21.15.23:0/2911338162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f80f6860 0x7f06f810c1c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f06e000e7c0 tx=0x7f06e000b040).stop 2021-11-19T08:40:23.813 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.817+0000 7f06fd0b8700 1 -- 172.21.15.23:0/2911338162 shutdown_connections 2021-11-19T08:40:23.813 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.817+0000 7f06fd0b8700 1 --2- 172.21.15.23:0/2911338162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06f80f55c0 0x7f06f8109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:23.813 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.817+0000 7f06fd0b8700 1 --2- 172.21.15.23:0/2911338162 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f06e405c2a0 0x7f06e405e760 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:23.813 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.817+0000 7f06fd0b8700 1 --2- 172.21.15.23:0/2911338162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06f80f6860 0x7f06f810c1c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.817+0000 7f06fd0b8700 1 -- 172.21.15.23:0/2911338162 >> 172.21.15.23:0/2911338162 conn(0x7f06f80f0d40 msgr2=0x7f06f80fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.817+0000 7f06fd0b8700 1 -- 172.21.15.23:0/2911338162 shutdown_connections 2021-11-19T08:40:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:23.818+0000 7f06fd0b8700 1 -- 172.21.15.23:0/2911338162 wait complete. 2021-11-19T08:40:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 2 2021-11-19T08:40:24.306 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-19T08:40:24.321 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi023.front.sepia.ceph.com 2021-11-19T08:40:24.321 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs set cephfs max_mds 2' 2021-11-19T08:40:24.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:24 smithi134 conmon[36363]: cluster 2021-11-19T08:40:22.825084+0000 mgr.smithi023.ehlpcl (mgr.14164) 188 : cluster 2021-11-19T08:40:24.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:24 smithi134 conmon[36363]: [DBG] pgmap v151: 65 pgs: 3 creating+activating, 11 active+clean, 51 unknown; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:24.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:24 smithi134 conmon[36363]: cluster 2021-11-19T08:40:23.222996+0000 mon.smithi023 (mon.0) 469 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2021-11-19T08:40:24.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:24 smithi134 conmon[36363]: audit 2021-11-19T08:40:23.815966+0000 mon.smithi023 (mon.0) 470 : 2021-11-19T08:40:24.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:24 smithi134 conmon[36363]: audit [DBG] from='client.? 172.21.15.23:0/2911338162' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-19T08:40:24.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:24 smithi023 conmon[33490]: cluster 2021-11-19T08:40:22.825084 2021-11-19T08:40:24.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:24 smithi023 conmon[33490]: +0000 mgr.smithi023.ehlpcl (mgr.14164) 188 : cluster [DBG] pgmap v151: 65 pgs: 3 creating+activating, 11 active+clean, 51 unknown; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:24.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:24 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:40:24.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:24 smithi023 conmon[33490]: 08:40:23.222996+0000 mon.smithi023 (mon.0) 469 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2021-11-19T08:40:24.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:24 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:40:24.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:24 smithi023 conmon[33490]: -19T08:40:23.815966+0000 mon.smithi023 (mon 2021-11-19T08:40:24.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:24 smithi023 conmon[33490]: .0) 470 : audit [DBG] from='client.? 172.21.15.23:0/2911338162' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-19T08:40:24.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:24 smithi023 conmon[33490]: 2021-11-19T08:40:25.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: audit 2021-11-19T08:40:24.442077+0000 mon.smithi023 ( 2021-11-19T08:40:25.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: mon.0) 471 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:25.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: audit 2021-11-19T08:40:24.442990+0000 mon.smithi023 (mon.0) 472 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.jqsnzo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-19T08:40:25.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: audit 2021-11-19T08:40:24.448620+0000 mon.smithi023 (mon.0) 473 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.jqsnzo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-19T08:40:25.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: audit 2021-11-19T08:40:24.449351+0000 mon.smithi023 (mon.0) 474 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:40:25.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: cephadm 2021-11-19T08:40:24.450188+0000 mgr.smithi023.ehlpcl (mgr.14164) 189 : cephadm [INF] Deploying daemon mds.cephfs.smithi134.jqsnzo on smithi134 2021-11-19T08:40:25.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: cluster 2021-11-19T08:40:25.222194+0000 mon.smithi023 (mon.0) 475 : cluster [DBG] mds.? [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] up:boot 2021-11-19T08:40:25.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: cluster 2021-11-19T08:40:25.222250+0000 mon.smithi023 (mon.0) 476 : cluster [INF] daemon mds.cephfs.smithi023.fuvzkr assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2021-11-19T08:40:25.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: cluster 2021-11-19T08:40:25.222301+0000 mon.smithi023 (mon.0) 477 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-19T08:40:25.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: cluster 2021-11-19T08:40:25.222328+0000 mon.smithi023 (mon.0) 478 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2021-11-19T08:40:25.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: cluster 2021-11-19T08:40:25.222340+0000 mon.smithi023 (mon.0) 479 : cluster [INF] Cluster is now healthy 2021-11-19T08:40:25.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: cluster 2021-11-19T08:40:25.222658+0000 mon.smithi023 (mon.0) 480 : cluster [DBG] fsmap cephfs:0 1 up:standby 2021-11-19T08:40:25.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: audit 2021-11-19T08:40:25.222741+0000 mon.smithi023 (mon.0) 481 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.fuvzkr"}]: dispatch 2021-11-19T08:40:25.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: cluster 2021-11-19T08:40:25.227042+0000 mon.smithi023 (mon.0) 482 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi023.fuvzkr=up:creating} 2021-11-19T08:40:25.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: cluster 2021-11-19T08:40:25.238783+0000 mon.smithi023 (mon.0) 483 : cluster [INF] 2021-11-19T08:40:25.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:25 smithi134 conmon[36363]: daemon mds.cephfs.smithi023.fuvzkr is now active in filesystem cephfs as rank 0 2021-11-19T08:40:25.693 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.707+0000 7efca2c5f700 1 -- 172.21.15.23:0/3257471873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc9c0f66e0 msgr2=0x7efc9c0f6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:25.694 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.707+0000 7efca2c5f700 1 --2- 172.21.15.23:0/3257471873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc9c0f66e0 0x7efc9c0f6b40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7efc8c004660 tx=0x7efc8c009b30).stop 2021-11-19T08:40:25.694 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.707+0000 7efca2c5f700 1 -- 172.21.15.23:0/3257471873 shutdown_connections 2021-11-19T08:40:25.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.707+0000 7efca2c5f700 1 --2- 172.21.15.23:0/3257471873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc9c0f66e0 0x7efc9c0f6b40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:25.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.707+0000 7efca2c5f700 1 --2- 172.21.15.23:0/3257471873 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c0f5440 0x7efc9c0f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:25.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.707+0000 7efca2c5f700 1 -- 172.21.15.23:0/3257471873 >> 172.21.15.23:0/3257471873 conn(0x7efc9c0f0ba0 msgr2=0x7efc9c0f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:25.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.707+0000 7efca2c5f700 1 -- 172.21.15.23:0/3257471873 shutdown_connections 2021-11-19T08:40:25.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.708+0000 7efca2c5f700 1 -- 172.21.15.23:0/3257471873 wait complete. 2021-11-19T08:40:25.696 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:25 smithi023 conmon[33490]: audit 2021-11-19T08:40:24.442077+0000 mon.smithi023 (mon.0) 471 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:25.696 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:25 smithi023 conmon[33490]: audit 2021-11-19T08:40:24.442990+0000 mon.smithi023 (mon.0) 472 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.jqsnzo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-19T08:40:25.697 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:25 smithi023 conmon[33490]: audit 2021-11-19T08:40:24.448620+0000 mon.smithi023 (mon.0) 473 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.jqsnzo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-19T08:40:25.697 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:25 smithi023 conmon[33490]: audit 2021-11-19T08:40:24.449351+0000 mon.smithi023 (mon.0) 474 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:40:25.697 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:25 smithi023 conmon[33490]: cephadm 2021-11-19T08:40:24.450188+0000 mgr.smithi023.ehlpcl (mgr.14164) 189 : cephadm [INF] Deploying daemon mds.cephfs.smithi134.jqsnzo on smithi134 2021-11-19T08:40:25.697 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:25 smithi023 conmon[33490]: cluster 2021-11-19T08:40:25.222194+0000 mon.smithi023 (mon.0) 475 : cluster [DBG] mds.? [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] up:boot 2021-11-19T08:40:25.698 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:25 smithi023 conmon[33490]: cluster 2021-11-19T08:40:25.222250+0000 mon.smithi023 (mon.0) 476 : cluster [INF] daemon mds.cephfs.smithi023.fuvzkr assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2021-11-19T08:40:25.698 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:25 smithi023 conmon[33490]: cluster 2021-11-19T08:40:25.222301+0000 mon.smithi023 (mon.0) 477 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-11-19T08:40:25.698 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:25 smithi023 conmon[33490]: cluster 2021-11-19T08:40:25.222328+0000 mon.smithi023 (mon.0) 478 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2021-11-19T08:40:25.698 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:25 smithi023 conmon[33490]: cluster 2021-11-19T08:40:25.222340+0000 mon.smithi023 (mon.0) 479 : cluster [INF] Cluster is now healthy 2021-11-19T08:40:25.699 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:25 smithi023 conmon[33490]: cluster 2021-11-19T08:40:25.222658+0000 mon.smithi023 (mon.0) 480 : cluster [DBG] fsmap cephfs:0 1 up:standby 2021-11-19T08:40:25.699 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:25 smithi023 conmon[33490]: audit 2021-11-19T08:40:25.222741+0000 mon.smithi023 (mon.0) 481 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.fuvzkr"}]: dispatch 2021-11-19T08:40:25.699 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:25 smithi023 conmon[33490]: cluster 2021-11-19T08:40:25.227042+0000 mon.smithi023 (mon.0) 482 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi023.fuvzkr=up:creating} 2021-11-19T08:40:25.699 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:25 smithi023 conmon[33490]: cluster 2021-11-19T08:40:25.238783+0000 mon.smithi023 (mon.0) 483 : cluster [INF] daemon mds.cephfs.smithi023.fuvzkr is now active in filesystem cephfs as rank 0 2021-11-19T08:40:25.703 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.708+0000 7efca2c5f700 1 Processor -- start 2021-11-19T08:40:25.703 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.708+0000 7efca2c5f700 1 -- start start 2021-11-19T08:40:25.703 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.708+0000 7efca2c5f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c0f5440 0x7efc9c10bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:25.704 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.708+0000 7efca2c5f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc9c0f66e0 0x7efc9c10e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:25.704 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.708+0000 7efca2c5f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c0602b0 con 0x7efc9c0f66e0 2021-11-19T08:40:25.704 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.708+0000 7efca2c5f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c0603f0 con 0x7efc9c0f5440 2021-11-19T08:40:25.704 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.709+0000 7efca1c5d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c0f5440 0x7efc9c10bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:25.705 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.708+0000 7efca145c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc9c0f66e0 0x7efc9c10e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:25.705 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.709+0000 7efca1c5d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c0f5440 0x7efc9c10bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.23:51586/0 (socket says 172.21.15.23:51586) 2021-11-19T08:40:25.705 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.709+0000 7efca1c5d700 1 -- 172.21.15.23:0/3352112131 learned_addr learned my addr 172.21.15.23:0/3352112131 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:25.705 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.709+0000 7efca1c5d700 1 -- 172.21.15.23:0/3352112131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc9c0f66e0 msgr2=0x7efc9c10e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:25.706 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.709+0000 7efca1c5d700 1 --2- 172.21.15.23:0/3352112131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc9c0f66e0 0x7efc9c10e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:25.706 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.709+0000 7efca1c5d700 1 -- 172.21.15.23:0/3352112131 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c005040 con 0x7efc9c0f5440 2021-11-19T08:40:25.706 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.709+0000 7efca145c700 1 --2- 172.21.15.23:0/3352112131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc9c0f66e0 0x7efc9c10e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-11-19T08:40:25.706 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.709+0000 7efca1c5d700 1 --2- 172.21.15.23:0/3352112131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c0f5440 0x7efc9c10bdf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7efc9800e7c0 tx=0x7efc9800b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.709+0000 7efc927fc700 1 -- 172.21.15.23:0/3352112131 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc980092e0 con 0x7efc9c0f5440 2021-11-19T08:40:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.709+0000 7efca2c5f700 1 -- 172.21.15.23:0/3352112131 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9c10e7e0 con 0x7efc9c0f5440 2021-11-19T08:40:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.709+0000 7efc927fc700 1 -- 172.21.15.23:0/3352112131 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc9800f040 con 0x7efc9c0f5440 2021-11-19T08:40:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.709+0000 7efc927fc700 1 -- 172.21.15.23:0/3352112131 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc980147c0 con 0x7efc9c0f5440 2021-11-19T08:40:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.709+0000 7efca2c5f700 1 -- 172.21.15.23:0/3352112131 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9c10ec60 con 0x7efc9c0f5440 2021-11-19T08:40:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.710+0000 7efca2c5f700 1 -- 172.21.15.23:0/3352112131 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc9c045ad0 con 0x7efc9c0f5440 2021-11-19T08:40:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.711+0000 7efc927fc700 1 -- 172.21.15.23:0/3352112131 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7efc9801b030 con 0x7efc9c0f5440 2021-11-19T08:40:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.711+0000 7efc927fc700 1 --2- 172.21.15.23:0/3352112131 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7efc8805c300 0x7efc8805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.711+0000 7efc927fc700 1 -- 172.21.15.23:0/3352112131 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7efc98075e50 con 0x7efc9c0f5440 2021-11-19T08:40:25.709 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.711+0000 7efca145c700 1 --2- 172.21.15.23:0/3352112131 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7efc8805c300 0x7efc8805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:25.709 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.711+0000 7efca145c700 1 --2- 172.21.15.23:0/3352112131 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7efc8805c300 0x7efc8805e7c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7efc8c005010 tx=0x7efc8c016040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:25.709 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.715+0000 7efc927fc700 1 -- 172.21.15.23:0/3352112131 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc98048b10 con 0x7efc9c0f5440 2021-11-19T08:40:25.899 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:25.914+0000 7efca2c5f700 1 -- 172.21.15.23:0/3352112131 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7efc9c0fdcd0 con 0x7efc9c0f5440 2021-11-19T08:40:26.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:26.228+0000 7efc927fc700 1 -- 172.21.15.23:0/3352112131 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v5) v1 ==== 105+0+0 (secure 0 0 0) 0x7efc98019070 con 0x7efc9c0f5440 2021-11-19T08:40:26.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:26.230+0000 7efca2c5f700 1 -- 172.21.15.23:0/3352112131 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7efc8805c300 msgr2=0x7efc8805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:26.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:26.230+0000 7efca2c5f700 1 --2- 172.21.15.23:0/3352112131 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7efc8805c300 0x7efc8805e7c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7efc8c005010 tx=0x7efc8c016040).stop 2021-11-19T08:40:26.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:26.230+0000 7efca2c5f700 1 -- 172.21.15.23:0/3352112131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c0f5440 msgr2=0x7efc9c10bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:26.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:26.230+0000 7efca2c5f700 1 --2- 172.21.15.23:0/3352112131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c0f5440 0x7efc9c10bdf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7efc9800e7c0 tx=0x7efc9800b040).stop 2021-11-19T08:40:26.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:26.230+0000 7efca2c5f700 1 -- 172.21.15.23:0/3352112131 shutdown_connections 2021-11-19T08:40:26.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:26.230+0000 7efca2c5f700 1 --2- 172.21.15.23:0/3352112131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c0f5440 0x7efc9c10bdf0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:26.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:26.230+0000 7efca2c5f700 1 --2- 172.21.15.23:0/3352112131 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7efc8805c300 0x7efc8805e7c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:26.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:26.230+0000 7efca2c5f700 1 --2- 172.21.15.23:0/3352112131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc9c0f66e0 0x7efc9c10e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:26.217 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:26.230+0000 7efca2c5f700 1 -- 172.21.15.23:0/3352112131 >> 172.21.15.23:0/3352112131 conn(0x7efc9c0f0ba0 msgr2=0x7efc9c0fa130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:26.217 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:26.230+0000 7efca2c5f700 1 -- 172.21.15.23:0/3352112131 shutdown_connections 2021-11-19T08:40:26.217 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:26.230+0000 7efca2c5f700 1 -- 172.21.15.23:0/3352112131 wait complete. 2021-11-19T08:40:26.651 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-19T08:40:26.664 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi023.front.sepia.ceph.com 2021-11-19T08:40:26.665 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2021-11-19T08:40:26.691 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: cluster 2021-11-19T08:40:24.825475+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:40:26.691 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: 14164) 190 : cluster [DBG] pgmap v153: 65 pgs: 3 creating+activating, 47 active+clean, 15 unknown; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:26.691 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: audit 2021-11-19T08:40:25.915964+0000 mon.smithi023 (mon.0) 484 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2021-11-19T08:40:26.692 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: audit 2021-11-19T08:40:25.915969+0000 mon.smithi134 (mon.1) 15 : audit [INF] from='client.? 172.21.15.23:0/3352112131' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2021-11-19T08:40:26.692 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: cluster 2021-11-19T08:40:26.225151+0000 mon.smithi023 (mon.0) 485 2021-11-19T08:40:26.692 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2021-11-19T08:40:26.693 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:40:26.693 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: 08:40:26.228895+0000 mon.smithi023 (mon.0 2021-11-19T08:40:26.693 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: ) 486 : cluster [DBG] mds.? [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] up:active 2021-11-19T08:40:26.694 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: audit 2021 2021-11-19T08:40:26.694 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: -11-19T08:40:26.228958+0000 mon.smithi023 2021-11-19T08:40:26.694 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: (mon.0) 487 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2021-11-19T08:40:26.694 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: cluster 2021-11- 2021-11-19T08:40:26.695 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: 19T08:40:26.229064+0000 mon.smithi023 (mon. 2021-11-19T08:40:26.695 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:26 smithi023 conmon[33490]: 0) 488 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi023.fuvzkr=up:active} 2021-11-19T08:40:26.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:26 smithi134 conmon[36363]: cluster 2021-11-19T08:40:24.825475+0000 mgr.smithi023.ehlpcl (mgr.14164) 190 : cluster [DBG] pgmap v153: 65 pgs: 3 creating+activating, 47 active+clean, 15 unknown; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:26.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:26 smithi134 conmon[36363]: audit 2021-11-19T08:40:25.915964+0000 mon.smithi023 (mon.0) 484 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2021-11-19T08:40:26.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:26 smithi134 conmon[36363]: audit 2021-11-19T08:40:25.915969+0000 mon.smithi134 (mon.1) 15 : audit [INF] from='client.? 172.21.15.23:0/3352112131' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2021-11-19T08:40:26.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:26 smithi134 conmon[36363]: cluster 2021-11-19T08:40:26.225151+0000 mon.smithi023 (mon.0) 485 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2021-11-19T08:40:26.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:26 smithi134 conmon[36363]: 2021-11-19T08:40:26.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:26 smithi134 conmon[36363]: cluster 2021-11-19T08:40 2021-11-19T08:40:26.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:26 smithi134 conmon[36363]: :26.228895+0000 mon.smithi023 (mon.0) 486 : cluster [DBG] mds.? [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] up:active 2021-11-19T08:40:26.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:26 smithi134 conmon[36363]: audit 2021-11-19T08: 2021-11-19T08:40:26.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:26 smithi134 conmon[36363]: 40:26.228958+0000 mon.smithi023 (mon.0) 487 : audit [INF] 2021-11-19T08:40:26.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:26 smithi134 conmon[36363]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2021-11-19T08:40:26.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:26 smithi134 conmon[36363]: cluster 2021-11- 2021-11-19T08:40:26.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:26 smithi134 conmon[36363]: 19T08:40:26.229064+0000 mon.smithi023 (mon.0) 488 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi023.fuvzkr=up:active} 2021-11-19T08:40:27.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:27 smithi134 conmon[36363]: audit 2021-11-19T08:40:26.519342+0000 mon.smithi023 (mon.0) 489 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:27.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:27 smithi134 conmon[36363]: audit 2021-11-19T08:40:26.520063+0000 2021-11-19T08:40:27.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:27 smithi134 conmon[36363]: mon.smithi023 (mon.0) 490 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi023.wazanl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-19T08:40:27.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:27 smithi134 conmon[36363]: audit 2021-11-19T08:40:26.522709+0000 mon.smithi023 (mon.0) 491 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi023.wazanl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-19T08:40:27.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:27 smithi134 conmon[36363]: audit 2021-11-19T08:40:26.523386+0000 mon.smithi023 (mon.0) 492 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:40:27.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:27 smithi134 conmon[36363]: cephadm 2021-11-19T08:40:26.524003+0000 mgr.smithi023.ehlpcl (mgr.14164) 191 : cephadm [INF] Deploying daemon mds.cephfs.smithi023.wazanl on smithi023 2021-11-19T08:40:27.829 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:27 smithi023 conmon[33490]: audit 2021-11-19T08:40:26.519342+0000 mon.smithi023 (mon.0) 489 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:27.830 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:27 smithi023 conmon[33490]: audit 2021-11-19T08:40:26.520063+0000 mon.smithi023 (mon.0) 490 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi023.wazanl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-19T08:40:27.830 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:27 smithi023 conmon[33490]: audit 2021-11-19T08:40:26.522709+0000 mon.smithi023 (mon.0) 491 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi023.wazanl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-19T08:40:27.830 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:27 smithi023 conmon[33490]: audit 2021-11-19T08:40:26.523386+0000 mon.smithi023 (mon.0) 492 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:40:27.830 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:27 smithi023 conmon[33490]: cephadm 2021-11-19T08:40:26.524003+0000 mgr.smithi023.ehlpcl (mgr.14164) 191 : cephadm [INF] Deploying daemon mds.cephfs.smithi023.wazanl on smithi023 2021-11-19T08:40:28.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.646+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/2830383263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d580f7d40 msgr2=0x7f2d580f8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:28.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.646+0000 7f2d5d7b0700 1 --2- 172.21.15.23:0/2830383263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d580f7d40 0x7f2d580f8160 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f2d480099e0 tx=0x7f2d48004d10).stop 2021-11-19T08:40:28.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.646+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/2830383263 shutdown_connections 2021-11-19T08:40:28.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.646+0000 7f2d5d7b0700 1 --2- 172.21.15.23:0/2830383263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d580f86e0 0x7f2d580f4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:28.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.646+0000 7f2d5d7b0700 1 --2- 172.21.15.23:0/2830383263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d580f7d40 0x7f2d580f8160 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:28.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.646+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/2830383263 >> 172.21.15.23:0/2830383263 conn(0x7f2d580f04c0 msgr2=0x7f2d580f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:28.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.646+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/2830383263 shutdown_connections 2021-11-19T08:40:28.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.646+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/2830383263 wait complete. 2021-11-19T08:40:28.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.647+0000 7f2d5d7b0700 1 Processor -- start 2021-11-19T08:40:28.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.647+0000 7f2d5d7b0700 1 -- start start 2021-11-19T08:40:28.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.647+0000 7f2d5d7b0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d580f7d40 0x7f2d581070f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:28.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.647+0000 7f2d5d7b0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d580f86e0 0x7f2d581075e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:28.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.647+0000 7f2d5d7b0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d58107b90 con 0x7f2d580f7d40 2021-11-19T08:40:28.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.647+0000 7f2d5d7b0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d58107cd0 con 0x7f2d580f86e0 2021-11-19T08:40:28.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.647+0000 7f2d56ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d580f7d40 0x7f2d581070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:28.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.647+0000 7f2d56ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d580f7d40 0x7f2d581070f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46946/0 (socket says 172.21.15.23:46946) 2021-11-19T08:40:28.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.647+0000 7f2d56ffd700 1 -- 172.21.15.23:0/4001478922 learned_addr learned my addr 172.21.15.23:0/4001478922 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:28.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.647+0000 7f2d567fc700 1 --2- 172.21.15.23:0/4001478922 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d580f86e0 0x7f2d581075e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:28.639 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.648+0000 7f2d56ffd700 1 -- 172.21.15.23:0/4001478922 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d580f86e0 msgr2=0x7f2d581075e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:28.639 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.648+0000 7f2d56ffd700 1 --2- 172.21.15.23:0/4001478922 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d580f86e0 0x7f2d581075e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:28.640 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.648+0000 7f2d56ffd700 1 -- 172.21.15.23:0/4001478922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d48009640 con 0x7f2d580f7d40 2021-11-19T08:40:28.640 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.648+0000 7f2d56ffd700 1 --2- 172.21.15.23:0/4001478922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d580f7d40 0x7f2d581070f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2d480078b0 tx=0x7f2d48004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:28.640 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.648+0000 7f2d4f7fe700 1 -- 172.21.15.23:0/4001478922 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d4800e030 con 0x7f2d580f7d40 2021-11-19T08:40:28.640 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.648+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/4001478922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d58108610 con 0x7f2d580f7d40 2021-11-19T08:40:28.641 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.648+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/4001478922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d58108af0 con 0x7f2d580f7d40 2021-11-19T08:40:28.641 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.648+0000 7f2d4f7fe700 1 -- 172.21.15.23:0/4001478922 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d48007d50 con 0x7f2d580f7d40 2021-11-19T08:40:28.642 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.648+0000 7f2d4f7fe700 1 -- 172.21.15.23:0/4001478922 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d48018070 con 0x7f2d580f7d40 2021-11-19T08:40:28.642 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.650+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/4001478922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d3c004fa0 con 0x7f2d580f7d40 2021-11-19T08:40:28.643 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.650+0000 7f2d4f7fe700 1 -- 172.21.15.23:0/4001478922 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2d4801d420 con 0x7f2d580f7d40 2021-11-19T08:40:28.644 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.650+0000 7f2d4f7fe700 1 --2- 172.21.15.23:0/4001478922 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2d4405c2b0 0x7f2d4405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:28.644 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.650+0000 7f2d4f7fe700 1 -- 172.21.15.23:0/4001478922 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f2d48077c20 con 0x7f2d580f7d40 2021-11-19T08:40:28.644 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.650+0000 7f2d567fc700 1 --2- 172.21.15.23:0/4001478922 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2d4405c2b0 0x7f2d4405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:28.645 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.651+0000 7f2d567fc700 1 --2- 172.21.15.23:0/4001478922 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2d4405c2b0 0x7f2d4405e770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f2d40004c40 tx=0x7f2d4000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:28.645 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.655+0000 7f2d4f7fe700 1 -- 172.21.15.23:0/4001478922 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d4804a8a0 con 0x7f2d580f7d40 2021-11-19T08:40:28.700 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:28 smithi023 conmon[33490]: cluster 2021-11-19T08:40:26.825972 2021-11-19T08:40:28.701 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:28 smithi023 conmon[33490]: +0000 mgr.smithi023.ehlpcl (mgr.14164) 192 : cluster [DBG] pgmap v154: 65 pgs: 65 active+clean; 0 B data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:28.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:28 smithi134 conmon[36363]: cluster 2021-11-19T08:40:26.825972+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:40:28.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:28 smithi134 conmon[36363]: 192 : cluster [DBG] pgmap v154: 65 pgs: 65 active+clean; 0 B data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:28.862+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/4001478922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f2d3c005bf0 con 0x7f2d580f7d40 2021-11-19T08:40:29.760 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:29.775+0000 7f2d4f7fe700 1 -- 172.21.15.23:0/4001478922 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v6) v1 ==== 121+0+0 (secure 0 0 0) 0x7f2d4804dec0 con 0x7f2d580f7d40 2021-11-19T08:40:29.762 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:29.777+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/4001478922 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2d4405c2b0 msgr2=0x7f2d4405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:29.762 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:29.777+0000 7f2d5d7b0700 1 --2- 172.21.15.23:0/4001478922 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2d4405c2b0 0x7f2d4405e770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f2d40004c40 tx=0x7f2d4000b040).stop 2021-11-19T08:40:29.763 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:29.777+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/4001478922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d580f7d40 msgr2=0x7f2d581070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:29.763 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:29.777+0000 7f2d5d7b0700 1 --2- 172.21.15.23:0/4001478922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d580f7d40 0x7f2d581070f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2d480078b0 tx=0x7f2d48004d10).stop 2021-11-19T08:40:29.763 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:29.777+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/4001478922 shutdown_connections 2021-11-19T08:40:29.763 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:29.777+0000 7f2d5d7b0700 1 --2- 172.21.15.23:0/4001478922 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2d4405c2b0 0x7f2d4405e770 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:29.763 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:29.777+0000 7f2d5d7b0700 1 --2- 172.21.15.23:0/4001478922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d580f7d40 0x7f2d581070f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:29.764 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:29.777+0000 7f2d5d7b0700 1 --2- 172.21.15.23:0/4001478922 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d580f86e0 0x7f2d581075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:29.764 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:29.777+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/4001478922 >> 172.21.15.23:0/4001478922 conn(0x7f2d580f04c0 msgr2=0x7f2d580fb5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:29.764 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:29.777+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/4001478922 shutdown_connections 2021-11-19T08:40:29.764 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:29.777+0000 7f2d5d7b0700 1 -- 172.21.15.23:0/4001478922 wait complete. 2021-11-19T08:40:29.942 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:29 smithi134 conmon[36363]: audit 2021-11-19T08:40:28.762230+0000 mon.smithi023 (mon.0) 493 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:29.943 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:29 smithi134 conmon[36363]: audit 2021-11-19T08:40:28.763103+0000 mon.smithi023 (mon.0) 494 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.opxllb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-19T08:40:29.943 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:29 smithi134 conmon[36363]: audit 2021-11-19T08:40:28.767962+0000 mon.smithi023 (mon.0) 495 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.opxllb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-19T08:40:29.943 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:29 smithi134 conmon[36363]: audit 2021-11-19T08:40:28.768724+0000 mon.smithi023 (mon.0) 496 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:40:29.944 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:29 smithi134 conmon[36363]: cephadm 2021-11-19T08:40:28.769373+0000 mgr.smithi023.ehlpcl (mgr.14164) 193 : cephadm [INF] Deploying daemon mds.cephfs.smithi134.opxllb on smithi134 2021-11-19T08:40:29.944 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:29 smithi134 conmon[36363]: audit 2021-11-19T08:40:28.863757+0000 mon.smithi023 (mon.0) 497 : audit [INF] from='client.? 172.21.15.23:0/4001478922' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2021-11-19T08:40:30.131 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:29 smithi023 conmon[33490]: audit 2021-11-19T08:40:28.762230+0000 mon.smithi023 (mon 2021-11-19T08:40:30.131 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:29 smithi023 conmon[33490]: .0) 493 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:30.131 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:29 smithi023 conmon[33490]: audit 2021-11-19T08:40:28.763103+0000 mon.smithi023 (mon.0) 494 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.opxllb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2021-11-19T08:40:30.132 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:29 smithi023 conmon[33490]: audit 2021-11-19T08:40:28.767962+0000 mon.smithi023 (mon.0) 495 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.opxllb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2021-11-19T08:40:30.132 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:29 smithi023 conmon[33490]: audit 2021-11-19T08:40:28.768724+0000 mon.smithi023 (mon.0) 496 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-11-19T08:40:30.132 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:29 smithi023 conmon[33490]: cephadm 2021-11-19T08:40:28.769373+0000 mgr.smithi023.ehlpcl (mgr.14164) 193 : cephadm [INF] Deploying daemon mds.cephfs.smithi134.opxllb on smithi134 2021-11-19T08:40:30.132 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:29 smithi023 conmon[33490]: audit 2021-11-19T08:40:28.863757+0000 mon.smithi023 (mon.0) 497 : audit [INF] from='client.? 172.21.15.23:0/4001478922' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2021-11-19T08:40:30.314 INFO:teuthology.run_tasks:Running task cephadm.shell... 2021-11-19T08:40:30.337 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi023.front.sepia.ceph.com 2021-11-19T08:40:30.337 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- bash -c 'ceph fs dump' 2021-11-19T08:40:31.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:30 smithi134 conmon[36363]: cluster 2021-11-19T08:40:28.826327+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:40:31.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:30 smithi134 conmon[36363]: ) 194 : cluster [DBG] pgmap v155: 65 pgs: 65 active+clean; 0 B data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:31.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:30 smithi134 conmon[36363]: audit 2021-11-19T08:40:29.775791+0000 mon.smithi023 (mon.0) 498 : audit [INF] from='client.? 172.21.15.23:0/4001478922' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2021-11-19T08:40:31.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:30 smithi134 conmon[36363]: cluster 2021-11-19T08:40:29.775899+0000 mon.smithi023 (mon.0) 499 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi023.fuvzkr=up:active} 2021-11-19T08:40:31.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:30 smithi134 conmon[36363]: audit 2021-11-19T08:40:30.746522+0000 mon.smithi023 (mon.0) 500 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:31.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:30 smithi134 conmon[36363]: audit 2021-11-19T08:40: 2021-11-19T08:40:31.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:30 smithi134 conmon[36363]: 30.747168+0000 mon.smithi023 (mon.0) 501 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:40:31.083 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:30 smithi023 conmon[33490]: cluster 2021-11-19T08:40:28.826327+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:40:31.083 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:30 smithi023 conmon[33490]: mgr.14164) 194 : cluster [DBG] pgmap v155: 65 pgs: 65 active+clean; 0 B data, 32 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:40:31.084 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:30 smithi023 conmon[33490]: audit 2021-11-19T08:40:29.775791+0000 mon.smithi023 (mon.0) 498 : audit [INF] from='client.? 172.21.15.23:0/4001478922' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2021-11-19T08:40:31.084 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:30 smithi023 conmon[33490]: cluster 2021-11-19T08:40:29.775899+0000 mon.smithi023 (mon.0) 499 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi023.fuvzkr=up:active} 2021-11-19T08:40:31.084 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:30 smithi023 conmon[33490]: audit 2021-11-19T08:40:30.746522+0000 mon.smithi023 (mon.0) 500 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:31.084 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:30 smithi023 conmon[33490]: audit 2021-11-19T08:40:30.747168+0000 mon.smithi023 (mon.0) 501 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:40:31.929 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.941+0000 7fe558f59700 1 -- 172.21.15.23:0/1862077937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5540f7980 msgr2=0x7fe5540f7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:31.929 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.941+0000 7fe558f59700 1 --2- 172.21.15.23:0/1862077937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5540f7980 0x7fe5540f7de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fe548004660 tx=0x7fe548009b30).stop 2021-11-19T08:40:31.929 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.941+0000 7fe558f59700 1 -- 172.21.15.23:0/1862077937 shutdown_connections 2021-11-19T08:40:31.930 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.941+0000 7fe558f59700 1 --2- 172.21.15.23:0/1862077937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5540f7980 0x7fe5540f7de0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:31.930 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.941+0000 7fe558f59700 1 --2- 172.21.15.23:0/1862077937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5540f66e0 0x7fe5540f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:31.930 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.941+0000 7fe558f59700 1 -- 172.21.15.23:0/1862077937 >> 172.21.15.23:0/1862077937 conn(0x7fe5540f1e40 msgr2=0x7fe5540f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:31.930 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.941+0000 7fe558f59700 1 -- 172.21.15.23:0/1862077937 shutdown_connections 2021-11-19T08:40:31.932 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.941+0000 7fe558f59700 1 -- 172.21.15.23:0/1862077937 wait complete. 2021-11-19T08:40:31.932 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.942+0000 7fe558f59700 1 Processor -- start 2021-11-19T08:40:31.932 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.942+0000 7fe558f59700 1 -- start start 2021-11-19T08:40:31.933 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.942+0000 7fe558f59700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5540f66e0 0x7fe554106f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:31.933 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.942+0000 7fe558f59700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5540f7980 0x7fe554107470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:31.933 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.942+0000 7fe558f59700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe554107a20 con 0x7fe5540f66e0 2021-11-19T08:40:31.933 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.942+0000 7fe558f59700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe554107b60 con 0x7fe5540f7980 2021-11-19T08:40:31.934 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.942+0000 7fe55259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5540f66e0 0x7fe554106f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:31.934 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.942+0000 7fe55259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5540f66e0 0x7fe554106f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46966/0 (socket says 172.21.15.23:46966) 2021-11-19T08:40:31.934 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.942+0000 7fe55259c700 1 -- 172.21.15.23:0/3462324797 learned_addr learned my addr 172.21.15.23:0/3462324797 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:31.934 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.943+0000 7fe551d9b700 1 --2- 172.21.15.23:0/3462324797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5540f7980 0x7fe554107470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:31.934 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.943+0000 7fe55259c700 1 -- 172.21.15.23:0/3462324797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5540f7980 msgr2=0x7fe554107470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:31.935 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.943+0000 7fe55259c700 1 --2- 172.21.15.23:0/3462324797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5540f7980 0x7fe554107470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:31.935 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.943+0000 7fe55259c700 1 -- 172.21.15.23:0/3462324797 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe548005040 con 0x7fe5540f66e0 2021-11-19T08:40:31.935 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.943+0000 7fe551d9b700 1 --2- 172.21.15.23:0/3462324797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5540f7980 0x7fe554107470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:40:31.935 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.943+0000 7fe55259c700 1 --2- 172.21.15.23:0/3462324797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5540f66e0 0x7fe554106f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fe544002700 tx=0x7fe544009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:31.936 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.943+0000 7fe542ffd700 1 -- 172.21.15.23:0/3462324797 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe544009320 con 0x7fe5540f66e0 2021-11-19T08:40:31.936 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.943+0000 7fe542ffd700 1 -- 172.21.15.23:0/3462324797 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe54400f040 con 0x7fe5540f66e0 2021-11-19T08:40:31.936 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.943+0000 7fe558f59700 1 -- 172.21.15.23:0/3462324797 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe554108500 con 0x7fe5540f66e0 2021-11-19T08:40:31.936 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.943+0000 7fe542ffd700 1 -- 172.21.15.23:0/3462324797 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe544014900 con 0x7fe5540f66e0 2021-11-19T08:40:31.937 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.943+0000 7fe558f59700 1 -- 172.21.15.23:0/3462324797 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe554108ac0 con 0x7fe5540f66e0 2021-11-19T08:40:31.937 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.944+0000 7fe558f59700 1 -- 172.21.15.23:0/3462324797 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe554045ad0 con 0x7fe5540f66e0 2021-11-19T08:40:31.938 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.944+0000 7fe542ffd700 1 -- 172.21.15.23:0/3462324797 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe544009480 con 0x7fe5540f66e0 2021-11-19T08:40:31.938 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.945+0000 7fe542ffd700 1 --2- 172.21.15.23:0/3462324797 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe53c05c2b0 0x7fe53c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:31.939 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.945+0000 7fe542ffd700 1 -- 172.21.15.23:0/3462324797 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fe544076210 con 0x7fe5540f66e0 2021-11-19T08:40:31.939 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.945+0000 7fe551d9b700 1 --2- 172.21.15.23:0/3462324797 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe53c05c2b0 0x7fe53c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:31.939 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.945+0000 7fe551d9b700 1 --2- 172.21.15.23:0/3462324797 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe53c05c2b0 0x7fe53c05e770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fe548009d60 tx=0x7fe548009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:31.939 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:31.949+0000 7fe542ffd700 1 -- 172.21.15.23:0/3462324797 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe544048ed0 con 0x7fe5540f66e0 2021-11-19T08:40:32.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:31 smithi134 conmon[36363]: cluster 2021-11-19T08:40:30.782602+0000 mon.smithi023 (mon.0) 502 : cluster [DBG] mds.? [v2:172.21.15.134:6824/1874302373,v1:172.21.15.134:6825/1874302373] up:boot 2021-11-19T08:40:32.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:31 smithi134 conmon[36363]: cluster 2021-11-19T08:40:30.782667+0000 mon.smithi023 (mon.0) 503 : cluster [DBG] mds.? [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987] up:boot 2021-11-19T08:40:32.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:31 smithi134 conmon[36363]: cluster 2021-11-19T08:40:30.782719+0000 mon.smithi023 (mon.0) 504 : cluster [INF] daemon mds.cephfs.smithi134.opxllb assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2021-11-19T08:40:32.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:31 smithi134 conmon[36363]: cluster 2021-11-19T08:40:30.782784+0000 mon.smithi023 (mon.0) 505 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2021-11-19T08:40:32.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:31 smithi134 conmon[36363]: cluster 2021-11-19T08:40:30.782800+0000 mon.smithi023 (mon.0) 506 : cluster [INF] Cluster is now healthy 2021-11-19T08:40:32.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:31 smithi134 conmon[36363]: cluster 2021-11-19T08:40:30.783071+0000 mon.smithi023 (mon.0) 507 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi023.fuvzkr=up:active} 2 up:standby 2021-11-19T08:40:32.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:31 smithi134 conmon[36363]: audit 2021-11-19T08:40:30.783160+0000 mon.smithi023 (mon.0) 508 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.jqsnzo"}]: dispatch 2021-11-19T08:40:32.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:31 smithi134 conmon[36363]: audit 2021-11-19T08:40:30.783664+0000 mon.smithi023 (mon.0) 509 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.opxllb"}]: dispatch 2021-11-19T08:40:32.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:31 smithi134 conmon[36363]: cluster 2021-11-19T08:40:30.785938+0000 mon.smithi023 (mon.0) 510 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi023.fuvzkr=up:active,1=cephfs.smithi134.opxllb=up:creating} 1 up:standby 2021-11-19T08:40:32.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:31 smithi134 conmon[36363]: cluster 2021-11-19T08:40:30.796731+0000 mon.smithi023 (mon.0) 511 : cluster [INF] daemon mds.cephfs.smithi134.opxllb is now active in filesystem cephfs as rank 1 2021-11-19T08:40:32.133 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:32.148+0000 7fe558f59700 1 -- 172.21.15.23:0/3462324797 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe5540feff0 con 0x7fe5540f66e0 2021-11-19T08:40:32.134 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:31 smithi023 conmon[33490]: cluster 2021-11-19T08:40:30.782602+0000 mon.smithi023 (mon.0) 502 : cluster [DBG] mds.? [v2:172.21.15.134:6824/1874302373,v1:172.21.15.134:6825/1874302373] up:boot 2021-11-19T08:40:32.134 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:31 smithi023 conmon[33490]: cluster 2021-11-19T08:40:30.782667+0000 mon.smithi023 (mon.0) 503 : cluster [DBG] mds.? [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987] up:boot 2021-11-19T08:40:32.135 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:31 smithi023 conmon[33490]: cluster 2021-11-19T08:40:30.782719+0000 mon.smithi023 (mon.0) 504 : cluster [INF] daemon mds.cephfs.smithi134.opxllb assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2021-11-19T08:40:32.135 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:31 smithi023 conmon[33490]: cluster 2021-11-19T08:40:30.782784+0000 mon.smithi023 (mon.0) 505 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2021-11-19T08:40:32.135 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:31 smithi023 conmon[33490]: cluster 2021-11-19T08:40:30.782800+0000 mon.smithi023 (mon.0) 506 : cluster [INF] Cluster is now healthy 2021-11-19T08:40:32.135 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:31 smithi023 conmon[33490]: cluster 2021-11-19T08:40:30.783071+0000 mon.smithi023 (mon.0) 507 : cluster 2021-11-19T08:40:32.136 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:31 smithi023 conmon[33490]: [DBG] fsmap cephfs:1 {0=cephfs.smithi023.fuvzkr=up:active} 2 up:standby 2021-11-19T08:40:32.136 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:31 smithi023 conmon[33490]: audit 2021-11-19T08:40:30.783160+0000 mon.smithi023 (mon.0) 508 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.jqsnzo"}]: dispatch 2021-11-19T08:40:32.136 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:31 smithi023 conmon[33490]: audit 2021-11-19T08:40:30.783664+0000 mon.smithi023 (mon.0) 509 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.opxllb"}]: dispatch 2021-11-19T08:40:32.136 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:31 smithi023 conmon[33490]: cluster 2021-11-19T08:40:30.785938+0000 mon.smithi023 (mon.0) 510 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi023.fuvzkr=up:active,1=cephfs.smithi134.opxllb=up:creating} 1 up:standby 2021-11-19T08:40:32.136 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:31 smithi023 conmon[33490]: cluster 2021-11-19T08:40:30.796731+0000 mon.smithi023 (mon.0) 511 : cluster [INF] daemon mds.cephfs.smithi134.opxllb is now active in filesystem cephfs as rank 1 2021-11-19T08:40:32.137 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:32.149+0000 7fe542ffd700 1 -- 172.21.15.23:0/3462324797 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 10 v10) v1 ==== 76+0+1455 (secure 0 0 0) 0x7fe54404c4f0 con 0x7fe5540f66e0 2021-11-19T08:40:32.137 INFO:teuthology.orchestra.run.smithi023.stdout:e10 2021-11-19T08:40:32.138 INFO:teuthology.orchestra.run.smithi023.stdout:enable_multiple, ever_enabled_multiple: 1,1 2021-11-19T08:40:32.138 INFO:teuthology.orchestra.run.smithi023.stdout:compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-19T08:40:32.138 INFO:teuthology.orchestra.run.smithi023.stdout:legacy client fscid: 1 2021-11-19T08:40:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:32.139 INFO:teuthology.orchestra.run.smithi023.stdout:Filesystem 'cephfs' (1) 2021-11-19T08:40:32.139 INFO:teuthology.orchestra.run.smithi023.stdout:fs_name cephfs 2021-11-19T08:40:32.139 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 10 2021-11-19T08:40:32.140 INFO:teuthology.orchestra.run.smithi023.stdout:flags 32 2021-11-19T08:40:32.140 INFO:teuthology.orchestra.run.smithi023.stdout:created 2021-11-19T08:40:21.201543+0000 2021-11-19T08:40:32.140 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2021-11-19T08:40:31.796123+0000 2021-11-19T08:40:32.140 INFO:teuthology.orchestra.run.smithi023.stdout:tableserver 0 2021-11-19T08:40:32.140 INFO:teuthology.orchestra.run.smithi023.stdout:root 0 2021-11-19T08:40:32.141 INFO:teuthology.orchestra.run.smithi023.stdout:session_timeout 60 2021-11-19T08:40:32.141 INFO:teuthology.orchestra.run.smithi023.stdout:session_autoclose 300 2021-11-19T08:40:32.141 INFO:teuthology.orchestra.run.smithi023.stdout:max_file_size 1099511627776 2021-11-19T08:40:32.141 INFO:teuthology.orchestra.run.smithi023.stdout:required_client_features {} 2021-11-19T08:40:32.142 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure 0 2021-11-19T08:40:32.142 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure_osd_epoch 0 2021-11-19T08:40:32.142 INFO:teuthology.orchestra.run.smithi023.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-11-19T08:40:32.142 INFO:teuthology.orchestra.run.smithi023.stdout:max_mds 2 2021-11-19T08:40:32.143 INFO:teuthology.orchestra.run.smithi023.stdout:in 0,1 2021-11-19T08:40:32.143 INFO:teuthology.orchestra.run.smithi023.stdout:up {0=14434,1=24283} 2021-11-19T08:40:32.143 INFO:teuthology.orchestra.run.smithi023.stdout:failed 2021-11-19T08:40:32.143 INFO:teuthology.orchestra.run.smithi023.stdout:damaged 2021-11-19T08:40:32.144 INFO:teuthology.orchestra.run.smithi023.stdout:stopped 2021-11-19T08:40:32.144 INFO:teuthology.orchestra.run.smithi023.stdout:data_pools [3] 2021-11-19T08:40:32.144 INFO:teuthology.orchestra.run.smithi023.stdout:metadata_pool 2 2021-11-19T08:40:32.144 INFO:teuthology.orchestra.run.smithi023.stdout:inline_data disabled 2021-11-19T08:40:32.144 INFO:teuthology.orchestra.run.smithi023.stdout:balancer 2021-11-19T08:40:32.145 INFO:teuthology.orchestra.run.smithi023.stdout:standby_count_wanted 1 2021-11-19T08:40:32.145 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.cephfs.smithi023.fuvzkr{0:14434} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646]] 2021-11-19T08:40:32.145 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.cephfs.smithi134.jqsnzo{0:24271} state up:standby-replay seq 2 addr [v2:172.21.15.134:6824/1874302373,v1:172.21.15.134:6825/1874302373]] 2021-11-19T08:40:32.145 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.cephfs.smithi134.opxllb{1:24283} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987]] 2021-11-19T08:40:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:32.147 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:32.151+0000 7fe558f59700 1 -- 172.21.15.23:0/3462324797 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe53c05c2b0 msgr2=0x7fe53c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:32.147 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:32.151+0000 7fe558f59700 1 --2- 172.21.15.23:0/3462324797 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe53c05c2b0 0x7fe53c05e770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fe548009d60 tx=0x7fe548009b30).stop 2021-11-19T08:40:32.147 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:32.151+0000 7fe558f59700 1 -- 172.21.15.23:0/3462324797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5540f66e0 msgr2=0x7fe554106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:32.148 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:32.151+0000 7fe558f59700 1 --2- 172.21.15.23:0/3462324797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5540f66e0 0x7fe554106f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fe544002700 tx=0x7fe544009d80).stop 2021-11-19T08:40:32.148 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:32.151+0000 7fe558f59700 1 -- 172.21.15.23:0/3462324797 shutdown_connections 2021-11-19T08:40:32.148 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:32.151+0000 7fe558f59700 1 --2- 172.21.15.23:0/3462324797 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe53c05c2b0 0x7fe53c05e770 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:32.148 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:32.151+0000 7fe558f59700 1 --2- 172.21.15.23:0/3462324797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5540f66e0 0x7fe554106f80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:32.149 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:32.151+0000 7fe558f59700 1 --2- 172.21.15.23:0/3462324797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5540f7980 0x7fe554107470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:32.149 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:32.151+0000 7fe558f59700 1 -- 172.21.15.23:0/3462324797 >> 172.21.15.23:0/3462324797 conn(0x7fe5540f1e40 msgr2=0x7fe5540fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:32.149 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:32.151+0000 7fe558f59700 1 -- 172.21.15.23:0/3462324797 shutdown_connections 2021-11-19T08:40:32.149 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:32.151+0000 7fe558f59700 1 -- 172.21.15.23:0/3462324797 wait complete. 2021-11-19T08:40:32.149 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 10 2021-11-19T08:40:32.574 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2021-11-19T08:40:33.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:32 smithi134 conmon[36363]: cluster 2021-11-19T08:40:30.826789+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:40:33.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:32 smithi134 conmon[36363]: mgr.14164) 195 : cluster [DBG] pgmap v156: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 6.5 KiB/s wr, 13 op/s 2021-11-19T08:40:33.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:32 smithi134 conmon[36363]: cluster 2021-11-19T08:40:31.796051+0000 mon.smithi023 (mon.0) 512 : cluster [DBG] mds.? [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987] up:active 2021-11-19T08:40:33.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:32 smithi134 conmon[36363]: cluster 2021-11-19T08:40:31.796409+0000 mon.smithi023 (mon.0) 513 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi023.fuvzkr=up:active,1=cephfs.smithi134.opxllb=up:active} 1 up:standby 2021-11-19T08:40:33.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:32 smithi134 conmon[36363]: cluster 2021-11-19T08:40:31.801113+0000 mon.smithi023 (mon 2021-11-19T08:40:33.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:32 smithi134 conmon[36363]: .0) 514 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi023.fuvzkr=up:active,1=cephfs.smithi134.opxllb=up:active} 1 up:standby-replay 2021-11-19T08:40:33.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:32 smithi134 conmon[36363]: audit 2021-11-19T08:40:32.150057+0000 mon.smithi023 (mon.0) 515 : audit 2021-11-19T08:40:33.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:32 smithi134 conmon[36363]: [DBG] from='client.? 172.21.15.23:0/3462324797' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-19T08:40:33.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:32 smithi023 conmon[33490]: cluster 2021-11-19T08:40:30.826789+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:40:33.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:32 smithi023 conmon[33490]: .14164) 195 : cluster [DBG] pgmap v156: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 6.5 KiB/s wr, 13 op/s 2021-11-19T08:40:33.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:32 smithi023 conmon[33490]: cluster 2021-11-19T08:40:31.796051+0000 mon.smithi023 (mon.0) 512 : cluster [DBG] mds.? [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987] up:active 2021-11-19T08:40:33.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:32 smithi023 conmon[33490]: cluster 2021-11-19T08:40:31.796409+0000 mon.smithi023 (mon.0) 513 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi023.fuvzkr=up:active,1=cephfs.smithi134.opxllb=up:active} 1 up:standby 2021-11-19T08:40:33.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:32 smithi023 conmon[33490]: cluster 2021-11-19T08:40:31.801113+0000 mon.smithi023 (mon.0) 514 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi023.fuvzkr=up:active,1=cephfs.smithi134.opxllb=up:active} 1 up:standby-replay 2021-11-19T08:40:33.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:32 smithi023 conmon[33490]: audit 2021-11-19T08:40:32.150057+0000 mon.smithi023 (mon.0) 515 : audit [DBG] from='client.? 172.21.15.23:0/3462324797' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2021-11-19T08:40:34.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:33 smithi134 conmon[36363]: cluster 2021-11-19T08:40:32.804284+0000 mon.smithi023 (mon.0) 516 : cluster [DBG] mds.? [v2:172.21.15.23:6828/2176063367,v1:172.21.15.23:6829/2176063367] up:boot 2021-11-19T08:40:34.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:33 smithi134 conmon[36363]: cluster 2021-11-19T08:40:32.804693+0000 mon.smithi023 (mon.0) 517 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi023.fuvzkr=up:active,1=cephfs.smithi134.opxllb=up:active} 1 up:standby-replay 1 up:standby 2021-11-19T08:40:34.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:33 smithi134 conmon[36363]: audit 2021-11-19T08:40:32.805210+0000 mon.smithi023 (mon. 2021-11-19T08:40:34.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:33 smithi134 conmon[36363]: 0) 518 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.wazanl"}]: dispatch 2021-11-19T08:40:34.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:33 smithi134 conmon[36363]: cluster 2021- 2021-11-19T08:40:34.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:33 smithi134 conmon[36363]: 11-19T08:40:32.807770+0000 mon.smithi023 (mon.0) 519 2021-11-19T08:40:34.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:33 smithi134 conmon[36363]: : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi023.fuvzkr=up:active,1=cephfs.smithi134.opxllb=up:active} 2 up:standby-replay 2021-11-19T08:40:34.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:33 smithi134 conmon[36363]: audit 2021-11-19T08 2021-11-19T08:40:34.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:33 smithi134 conmon[36363]: :40:33.620772+0000 mon.smithi023 (mon.0 2021-11-19T08:40:34.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:33 smithi134 conmon[36363]: ) 520 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:34.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:33 smithi023 conmon[33490]: cluster 2021-11-19T08:40:32.804284+0000 mon.smithi023 (mon.0) 516 : cluster [DBG] mds.? [v2:172.21.15.23:6828/2176063367,v1:172.21.15.23:6829/2176063367] up:boot 2021-11-19T08:40:34.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:33 smithi023 conmon[33490]: cluster 2021-11-19T08:40:32.804693+0000 mon.smithi023 (mon.0) 517 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi023.fuvzkr=up:active,1=cephfs.smithi134.opxllb=up:active} 1 up:standby-replay 1 up:standby 2021-11-19T08:40:34.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:33 smithi023 conmon[33490]: audit 2021-11-19T08:40:32.805210+0000 mon.smithi023 (mon.0) 518 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.wazanl"}]: dispatch 2021-11-19T08:40:34.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:33 smithi023 conmon[33490]: cluster 2021-11-19T08:40:32.807770+0000 mon.smithi023 (mon.0) 519 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi023.fuvzkr=up:active,1=cephfs.smithi134.opxllb=up:active} 2 up:standby-replay 2021-11-19T08:40:34.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:33 smithi023 conmon[33490]: audit 2021-11-19T08:40:33.620772+0000 mon.smithi023 (mon.0) 520 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:34.446 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.456+0000 7fd1ee681700 1 -- 172.21.15.23:0/2297755130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1e80f7d00 msgr2=0x7fd1e80f8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:34.447 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.456+0000 7fd1ee681700 1 --2- 172.21.15.23:0/2297755130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1e80f7d00 0x7fd1e80f8120 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fd1d8004660 tx=0x7fd1d8009b30).stop 2021-11-19T08:40:34.448 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.456+0000 7fd1ee681700 1 -- 172.21.15.23:0/2297755130 shutdown_connections 2021-11-19T08:40:34.448 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.456+0000 7fd1ee681700 1 --2- 172.21.15.23:0/2297755130 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1e80f8610 0x7fd1e80f4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:34.448 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.456+0000 7fd1ee681700 1 --2- 172.21.15.23:0/2297755130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1e80f7d00 0x7fd1e80f8120 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:34.449 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.456+0000 7fd1ee681700 1 -- 172.21.15.23:0/2297755130 >> 172.21.15.23:0/2297755130 conn(0x7fd1e80f0540 msgr2=0x7fd1e80f2960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:34.449 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.456+0000 7fd1ee681700 1 -- 172.21.15.23:0/2297755130 shutdown_connections 2021-11-19T08:40:34.450 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.456+0000 7fd1ee681700 1 -- 172.21.15.23:0/2297755130 wait complete. 2021-11-19T08:40:34.450 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.457+0000 7fd1ee681700 1 Processor -- start 2021-11-19T08:40:34.450 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.457+0000 7fd1ee681700 1 -- start start 2021-11-19T08:40:34.451 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.457+0000 7fd1ee681700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1e80f7d00 0x7fd1e80f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:34.451 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.457+0000 7fd1ee681700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1e80f8610 0x7fd1e80f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:34.451 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.457+0000 7fd1ee681700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80f5fb0 con 0x7fd1e80f7d00 2021-11-19T08:40:34.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.457+0000 7fd1ee681700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80f60f0 con 0x7fd1e80f8610 2021-11-19T08:40:34.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.457+0000 7fd1e7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1e80f7d00 0x7fd1e80f75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:34.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.458+0000 7fd1e7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1e80f7d00 0x7fd1e80f75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46984/0 (socket says 172.21.15.23:46984) 2021-11-19T08:40:34.453 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.458+0000 7fd1e7fff700 1 -- 172.21.15.23:0/1142606002 learned_addr learned my addr 172.21.15.23:0/1142606002 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:34.453 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.458+0000 7fd1e77fe700 1 --2- 172.21.15.23:0/1142606002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1e80f8610 0x7fd1e80f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:34.453 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.458+0000 7fd1e7fff700 1 -- 172.21.15.23:0/1142606002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1e80f8610 msgr2=0x7fd1e80f5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:34.454 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.458+0000 7fd1e7fff700 1 --2- 172.21.15.23:0/1142606002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1e80f8610 0x7fd1e80f5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:34.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.458+0000 7fd1e7fff700 1 -- 172.21.15.23:0/1142606002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d8005040 con 0x7fd1e80f7d00 2021-11-19T08:40:34.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.458+0000 7fd1e7fff700 1 --2- 172.21.15.23:0/1142606002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1e80f7d00 0x7fd1e80f75d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fd1d8000c00 tx=0x7fd1d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:34.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.458+0000 7fd1e4ff9700 1 -- 172.21.15.23:0/1142606002 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1d8019070 con 0x7fd1e80f7d00 2021-11-19T08:40:34.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.458+0000 7fd1ee681700 1 -- 172.21.15.23:0/1142606002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e80f6370 con 0x7fd1e80f7d00 2021-11-19T08:40:34.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.458+0000 7fd1e4ff9700 1 -- 172.21.15.23:0/1142606002 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1d8007e70 con 0x7fd1e80f7d00 2021-11-19T08:40:34.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.458+0000 7fd1e4ff9700 1 -- 172.21.15.23:0/1142606002 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1d8014c90 con 0x7fd1e80f7d00 2021-11-19T08:40:34.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.458+0000 7fd1ee681700 1 -- 172.21.15.23:0/1142606002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e80f68d0 con 0x7fd1e80f7d00 2021-11-19T08:40:34.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.459+0000 7fd1ee681700 1 -- 172.21.15.23:0/1142606002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e8045ad0 con 0x7fd1e80f7d00 2021-11-19T08:40:34.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.460+0000 7fd1e4ff9700 1 -- 172.21.15.23:0/1142606002 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd1d8025070 con 0x7fd1e80f7d00 2021-11-19T08:40:34.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.460+0000 7fd1e4ff9700 1 --2- 172.21.15.23:0/1142606002 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd1d005c2b0 0x7fd1d005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:34.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.460+0000 7fd1e4ff9700 1 -- 172.21.15.23:0/1142606002 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fd1d8076f20 con 0x7fd1e80f7d00 2021-11-19T08:40:34.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.460+0000 7fd1e77fe700 1 --2- 172.21.15.23:0/1142606002 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd1d005c2b0 0x7fd1d005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:34.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.460+0000 7fd1e77fe700 1 --2- 172.21.15.23:0/1142606002 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd1d005c2b0 0x7fd1d005e770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fd1dc009fd0 tx=0x7fd1dc00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:34.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.464+0000 7fd1e4ff9700 1 -- 172.21.15.23:0/1142606002 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1d80a32b0 con 0x7fd1e80f7d00 2021-11-19T08:40:34.649 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.664+0000 7fd1ee681700 1 -- 172.21.15.23:0/1142606002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1e8027a00 con 0x7fd1e80f7d00 2021-11-19T08:40:34.650 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.665+0000 7fd1e4ff9700 1 -- 172.21.15.23:0/1142606002 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2964 (secure 0 0 0) 0x7fd1e8027a00 con 0x7fd1e80f7d00 2021-11-19T08:40:34.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.667+0000 7fd1ee681700 1 -- 172.21.15.23:0/1142606002 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd1d005c2b0 msgr2=0x7fd1d005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:34.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.667+0000 7fd1ee681700 1 --2- 172.21.15.23:0/1142606002 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd1d005c2b0 0x7fd1d005e770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fd1dc009fd0 tx=0x7fd1dc00b040).stop 2021-11-19T08:40:34.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.667+0000 7fd1ee681700 1 -- 172.21.15.23:0/1142606002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1e80f7d00 msgr2=0x7fd1e80f75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:34.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.667+0000 7fd1ee681700 1 --2- 172.21.15.23:0/1142606002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1e80f7d00 0x7fd1e80f75d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fd1d8000c00 tx=0x7fd1d800b040).stop 2021-11-19T08:40:34.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.667+0000 7fd1ee681700 1 -- 172.21.15.23:0/1142606002 shutdown_connections 2021-11-19T08:40:34.654 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.667+0000 7fd1ee681700 1 --2- 172.21.15.23:0/1142606002 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd1d005c2b0 0x7fd1d005e770 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:34.654 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.667+0000 7fd1ee681700 1 --2- 172.21.15.23:0/1142606002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1e80f7d00 0x7fd1e80f75d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:34.654 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.667+0000 7fd1ee681700 1 --2- 172.21.15.23:0/1142606002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1e80f8610 0x7fd1e80f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:34.654 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.667+0000 7fd1ee681700 1 -- 172.21.15.23:0/1142606002 >> 172.21.15.23:0/1142606002 conn(0x7fd1e80f0540 msgr2=0x7fd1e80fb590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:34.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.667+0000 7fd1ee681700 1 -- 172.21.15.23:0/1142606002 shutdown_connections 2021-11-19T08:40:34.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:34.667+0000 7fd1ee681700 1 -- 172.21.15.23:0/1142606002 wait complete. 2021-11-19T08:40:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 12 2021-11-19T08:40:34.663 INFO:teuthology.orchestra.run.smithi023.stdout:true 2021-11-19T08:40:35.120 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:34 smithi023 conmon[33490]: cluster 2021-11-19T08:40:32.827190+0000 mgr.smithi023.ehlpcl (mgr.14164) 196 2021-11-19T08:40:35.121 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:34 smithi023 conmon[33490]: : cluster [DBG] pgmap v157: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s wr, 9 op/s 2021-11-19T08:40:35.121 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:34 smithi023 conmon[33490]: audit 2021-11-19T08:40:34.665927+0000 mon.smithi023 (mon.0) 521 : audit [DBG] from='client.? 172.21.15.23:0/1142606002' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:40:35.130 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c -- bash -c 'ceph --format=json mds versions | jq -e ". | add == 4"' 2021-11-19T08:40:35.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:34 smithi134 conmon[36363]: cluster 2021-11-19T08:40:32.827190+0000 mgr.smithi023.ehlpcl (mgr.14164) 196 : cluster [DBG] 2021-11-19T08:40:35.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:34 smithi134 conmon[36363]: pgmap v157: 65 pgs: 65 active+clean; 2.3 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 4.7 KiB/s wr, 9 op/s 2021-11-19T08:40:35.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:34 smithi134 conmon[36363]: audit 2021-11-19T08:40:34.665927+0000 mon.smithi023 (mon.0) 521 : audit [DBG] from='client.? 172.21.15.23:0/1142606002' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:40:36.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:36 smithi023 conmon[33490]: cluster 2021-11-19T08:40:34.827846+0000 mgr.smithi023.ehlpcl 2021-11-19T08:40:36.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:36 smithi023 conmon[33490]: (mgr.14164) 197 : cluster [DBG] pgmap v158: 65 pgs: 65 active+clean; 3.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 3.8 KiB/s wr, 9 op/s 2021-11-19T08:40:36.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:36 smithi023 conmon[33490]: cluster 2021-11-19T08:40:35.178395+0000 mon.smithi023 (mon 2021-11-19T08:40:36.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:36 smithi023 conmon[33490]: .0) 522 : cluster [DBG] mds.? [v2:172.21.15.134:6824/1874302373,v1:172.21.15.134:6825/1874302373] up:standby-replay 2021-11-19T08:40:36.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:36 smithi023 conmon[33490]: cluster 2021-11-19T08:40:36.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:36 smithi023 conmon[33490]: 2021-11-19T08:40:35.178462+0000 mon.smithi023 (mon.0 2021-11-19T08:40:36.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:36 smithi023 conmon[33490]: ) 523 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi023.fuvzkr=up:active,1=cephfs.smithi134.opxllb=up:active} 2 up:standby-replay 2021-11-19T08:40:36.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:36 smithi134 conmon[36363]: cluster 2021-11-19T08:40:34.827846+0000 mgr.smithi023.ehlpcl (mgr.14164) 197 : cluster [DBG] pgmap v158: 65 pgs: 65 active+clean; 3.5 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 3.8 KiB/s wr, 9 op/s 2021-11-19T08:40:36.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:36 smithi134 conmon[36363]: cluster 2021-11-19T08:40:35.178395+0000 2021-11-19T08:40:36.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:36 smithi134 conmon[36363]: mon.smithi023 (mon.0) 522 : cluster [DBG] mds.? [v2:172.21.15.134:6824/1874302373,v1:172.21.15.134:6825/1874302373] up:standby-replay 2021-11-19T08:40:36.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:36 smithi134 conmon[36363]: cluster 2021-11-19T08:40:35.178462+0000 mon.smithi023 (mon.0) 523 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi023.fuvzkr=up:active,1=cephfs.smithi134.opxllb=up:active} 2 up:standby-replay 2021-11-19T08:40:36.828 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.841+0000 7f62d48c9700 1 -- 172.21.15.23:0/1101135159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62d00f77d0 msgr2=0x7f62d00f7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:36.828 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.841+0000 7f62d48c9700 1 --2- 172.21.15.23:0/1101135159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62d00f77d0 0x7f62d00f7c30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f62c4004660 tx=0x7f62c4009b30).stop 2021-11-19T08:40:36.829 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.841+0000 7f62d48c9700 1 -- 172.21.15.23:0/1101135159 shutdown_connections 2021-11-19T08:40:36.829 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.841+0000 7f62d48c9700 1 --2- 172.21.15.23:0/1101135159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62d00f77d0 0x7f62d00f7c30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:36.829 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.841+0000 7f62d48c9700 1 --2- 172.21.15.23:0/1101135159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62d00f6530 0x7f62d00f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-19T08:40:36.830 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.841+0000 7f62d48c9700 1 -- 172.21.15.23:0/1101135159 >> 172.21.15.23:0/1101135159 conn(0x7f62d00f1cd0 msgr2=0x7f62d00f40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:36.830 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.841+0000 7f62d48c9700 1 -- 172.21.15.23:0/1101135159 shutdown_connections 2021-11-19T08:40:36.830 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.841+0000 7f62d48c9700 1 -- 172.21.15.23:0/1101135159 wait complete. 2021-11-19T08:40:36.830 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.842+0000 7f62d48c9700 1 Processor -- start 2021-11-19T08:40:36.830 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.842+0000 7f62d48c9700 1 -- start start 2021-11-19T08:40:36.831 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.842+0000 7f62d48c9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62d00f6530 0x7f62d0106da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:36.831 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.842+0000 7f62d48c9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62d00f77d0 0x7f62d0107290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:36.831 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.842+0000 7f62d48c9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d0107920 con 0x7f62d00f77d0 2021-11-19T08:40:36.831 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.842+0000 7f62d48c9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d01080d0 con 0x7f62d00f6530 2021-11-19T08:40:36.832 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.842+0000 7f62ce59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62d00f77d0 0x7f62d0107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:36.832 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.842+0000 7f62ce59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62d00f77d0 0x7f62d0107290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46994/0 (socket says 172.21.15.23:46994) 2021-11-19T08:40:36.832 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.842+0000 7f62ce59c700 1 -- 172.21.15.23:0/611636844 learned_addr learned my addr 172.21.15.23:0/611636844 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:36.832 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.842+0000 7f62ced9d700 1 --2- 172.21.15.23:0/611636844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62d00f6530 0x7f62d0106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:36.832 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.843+0000 7f62ce59c700 1 -- 172.21.15.23:0/611636844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62d00f6530 msgr2=0x7f62d0106da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:36.833 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.843+0000 7f62ce59c700 1 --2- 172.21.15.23:0/611636844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62d00f6530 0x7f62d0106da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:36.833 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.843+0000 7f62ce59c700 1 -- 172.21.15.23:0/611636844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c4005040 con 0x7f62d00f77d0 2021-11-19T08:40:36.833 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.843+0000 7f62ce59c700 1 --2- 172.21.15.23:0/611636844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62d00f77d0 0x7f62d0107290 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f62c4007a40 tx=0x7f62c4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:36.833 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.843+0000 7f62b77fe700 1 -- 172.21.15.23:0/611636844 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62c4018070 con 0x7f62d00f77d0 2021-11-19T08:40:36.834 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.843+0000 7f62d48c9700 1 -- 172.21.15.23:0/611636844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d0108350 con 0x7f62d00f77d0 2021-11-19T08:40:36.834 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.843+0000 7f62b77fe700 1 -- 172.21.15.23:0/611636844 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62c4013410 con 0x7f62d00f77d0 2021-11-19T08:40:36.834 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.843+0000 7f62b77fe700 1 -- 172.21.15.23:0/611636844 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62c401c620 con 0x7f62d00f77d0 2021-11-19T08:40:36.834 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.843+0000 7f62d48c9700 1 -- 172.21.15.23:0/611636844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d01088b0 con 0x7f62d00f77d0 2021-11-19T08:40:36.835 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.844+0000 7f62d48c9700 1 -- 172.21.15.23:0/611636844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d0059100 con 0x7f62d00f77d0 2021-11-19T08:40:36.836 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.845+0000 7f62b77fe700 1 -- 172.21.15.23:0/611636844 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f62c401c8e0 con 0x7f62d00f77d0 2021-11-19T08:40:36.836 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.845+0000 7f62b77fe700 1 --2- 172.21.15.23:0/611636844 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b805c2f0 0x7f62b805e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:36.836 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.845+0000 7f62b77fe700 1 -- 172.21.15.23:0/611636844 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f62c4077e10 con 0x7f62d00f77d0 2021-11-19T08:40:36.836 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.845+0000 7f62ced9d700 1 --2- 172.21.15.23:0/611636844 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b805c2f0 0x7f62b805e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:36.837 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.846+0000 7f62ced9d700 1 --2- 172.21.15.23:0/611636844 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b805c2f0 0x7f62b805e7b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f62d0107b60 tx=0x7f62c0008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:36.837 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:36.849+0000 7f62b77fe700 1 -- 172.21.15.23:0/611636844 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62c404aa70 con 0x7f62d00f77d0 2021-11-19T08:40:37.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.082+0000 7f62d48c9700 1 -- 172.21.15.23:0/611636844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f62d00fed60 con 0x7f62d00f77d0 2021-11-19T08:40:37.068 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.083+0000 7f62b77fe700 1 -- 172.21.15.23:0/611636844 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v13) v1 ==== 78+0+86 (secure 0 0 0) 0x7f62c404e090 con 0x7f62d00f77d0 2021-11-19T08:40:37.069 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.084+0000 7f62d48c9700 1 -- 172.21.15.23:0/611636844 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b805c2f0 msgr2=0x7f62b805e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:37.070 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.084+0000 7f62d48c9700 1 --2- 172.21.15.23:0/611636844 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b805c2f0 0x7f62b805e7b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f62d0107b60 tx=0x7f62c0008040).stop 2021-11-19T08:40:37.070 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.084+0000 7f62d48c9700 1 -- 172.21.15.23:0/611636844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62d00f77d0 msgr2=0x7f62d0107290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:37.070 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.084+0000 7f62d48c9700 1 --2- 172.21.15.23:0/611636844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62d00f77d0 0x7f62d0107290 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f62c4007a40 tx=0x7f62c4009b30).stop 2021-11-19T08:40:37.070 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.084+0000 7f62d48c9700 1 -- 172.21.15.23:0/611636844 shutdown_connections 2021-11-19T08:40:37.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.084+0000 7f62d48c9700 1 --2- 172.21.15.23:0/611636844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62d00f6530 0x7f62d0106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:37.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.084+0000 7f62d48c9700 1 --2- 172.21.15.23:0/611636844 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b805c2f0 0x7f62b805e7b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:37.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.084+0000 7f62d48c9700 1 --2- 172.21.15.23:0/611636844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62d00f77d0 0x7f62d0107290 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:37.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.084+0000 7f62d48c9700 1 -- 172.21.15.23:0/611636844 >> 172.21.15.23:0/611636844 conn(0x7f62d00f1cd0 msgr2=0x7f62d00fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:37.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.085+0000 7f62d48c9700 1 -- 172.21.15.23:0/611636844 shutdown_connections 2021-11-19T08:40:37.072 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.085+0000 7f62d48c9700 1 -- 172.21.15.23:0/611636844 wait complete. 2021-11-19T08:40:37.080 INFO:teuthology.orchestra.run.smithi023.stdout:true 2021-11-19T08:40:37.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:37 smithi023 conmon[33490]: audit 2021-11-19T08:40:37.083754+0000 mon.smithi023 (mon.0) 524 : audit [DBG] from='client.? 172.21.15.23:0/611636844' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2021-11-19T08:40:37.462 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2021-11-19T08:40:37.477 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-19T08:40:37.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:37 smithi134 conmon[36363]: audit 2021-11-19T08:40:37.083754+0000 mon.smithi023 (mon.0) 524 : audit [DBG] from='client.? 172.21.15.23:0/611636844' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2021-11-19T08:40:37.645 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.656+0000 7f5030b7b700 1 -- 172.21.15.23:0/4126592933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f502c0f9760 msgr2=0x7f502c0fbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:37.645 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.656+0000 7f5030b7b700 1 --2- 172.21.15.23:0/4126592933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f502c0f9760 0x7f502c0fbb90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f501c004660 tx=0x7f501c009b30).stop 2021-11-19T08:40:37.645 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.656+0000 7f5030b7b700 1 -- 172.21.15.23:0/4126592933 shutdown_connections 2021-11-19T08:40:37.646 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.656+0000 7f5030b7b700 1 --2- 172.21.15.23:0/4126592933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f502c0f9760 0x7f502c0fbb90 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:37.646 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.656+0000 7f5030b7b700 1 --2- 172.21.15.23:0/4126592933 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f502c0603b0 0x7f502c0f9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:37.646 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.656+0000 7f5030b7b700 1 -- 172.21.15.23:0/4126592933 >> 172.21.15.23:0/4126592933 conn(0x7f502c0f0c30 msgr2=0x7f502c0f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:37.647 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.656+0000 7f5030b7b700 1 -- 172.21.15.23:0/4126592933 shutdown_connections 2021-11-19T08:40:37.647 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.656+0000 7f5030b7b700 1 -- 172.21.15.23:0/4126592933 wait complete. 2021-11-19T08:40:37.647 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.656+0000 7f5030b7b700 1 Processor -- start 2021-11-19T08:40:37.647 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.656+0000 7f5030b7b700 1 -- start start 2021-11-19T08:40:37.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f5030b7b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f502c0603b0 0x7f502c106dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:37.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f5030b7b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f502c0f9760 0x7f502c1072c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:37.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f5030b7b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502c1078c0 con 0x7f502c0603b0 2021-11-19T08:40:37.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f5030b7b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502c108100 con 0x7f502c0f9760 2021-11-19T08:40:37.649 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f502b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f502c0603b0 0x7f502c106dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:37.649 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f502b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f502c0603b0 0x7f502c106dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47002/0 (socket says 172.21.15.23:47002) 2021-11-19T08:40:37.649 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f502b7fe700 1 -- 172.21.15.23:0/2568393709 learned_addr learned my addr 172.21.15.23:0/2568393709 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:37.649 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f502affd700 1 --2- 172.21.15.23:0/2568393709 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f502c0f9760 0x7f502c1072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:37.650 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f502b7fe700 1 -- 172.21.15.23:0/2568393709 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f502c0f9760 msgr2=0x7f502c1072c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:37.650 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f502b7fe700 1 --2- 172.21.15.23:0/2568393709 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f502c0f9760 0x7f502c1072c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:37.650 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f502b7fe700 1 -- 172.21.15.23:0/2568393709 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501c005040 con 0x7f502c0603b0 2021-11-19T08:40:37.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f502b7fe700 1 --2- 172.21.15.23:0/2568393709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f502c0603b0 0x7f502c106dd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f50140026d0 tx=0x7f5014009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:37.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f5023fff700 1 -- 172.21.15.23:0/2568393709 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50140079c0 con 0x7f502c0603b0 2021-11-19T08:40:37.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f5023fff700 1 -- 172.21.15.23:0/2568393709 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f501400f040 con 0x7f502c0603b0 2021-11-19T08:40:37.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f5030b7b700 1 -- 172.21.15.23:0/2568393709 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f502c1083e0 con 0x7f502c0603b0 2021-11-19T08:40:37.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f5023fff700 1 -- 172.21.15.23:0/2568393709 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50140148b0 con 0x7f502c0603b0 2021-11-19T08:40:37.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.657+0000 7f5030b7b700 1 -- 172.21.15.23:0/2568393709 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f502c1088f0 con 0x7f502c0603b0 2021-11-19T08:40:37.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.658+0000 7f5030b7b700 1 -- 172.21.15.23:0/2568393709 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f502c0f48c0 con 0x7f502c0603b0 2021-11-19T08:40:37.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.659+0000 7f5023fff700 1 -- 172.21.15.23:0/2568393709 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5014014a10 con 0x7f502c0603b0 2021-11-19T08:40:37.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.659+0000 7f5023fff700 1 --2- 172.21.15.23:0/2568393709 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f501805c300 0x7f501805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:37.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.659+0000 7f5023fff700 1 -- 172.21.15.23:0/2568393709 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f5014075e40 con 0x7f502c0603b0 2021-11-19T08:40:37.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.659+0000 7f502affd700 1 --2- 172.21.15.23:0/2568393709 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f501805c300 0x7f501805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:37.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.659+0000 7f502affd700 1 --2- 172.21.15.23:0/2568393709 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f501805c300 0x7f501805e7c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f501c009f50 tx=0x7f501c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:37.654 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.663+0000 7f5023fff700 1 -- 172.21.15.23:0/2568393709 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5014048b00 con 0x7f502c0603b0 2021-11-19T08:40:37.810 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.825+0000 7f5030b7b700 1 -- 172.21.15.23:0/2568393709 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f502c0fedd0 con 0x7f502c0603b0 2021-11-19T08:40:37.812 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.827+0000 7f5023fff700 1 -- 172.21.15.23:0/2568393709 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+14093 (secure 0 0 0) 0x7f501404c120 con 0x7f502c0603b0 2021-11-19T08:40:37.812 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:37.812 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":41,"fsid":"908b41ee-4913-11ec-8c2c-001a4aab830c","created":"2021-11-19T08:35:14.952900+0000","modified":"2021-11-19T08:40:23.207735+0000","last_up_change":"2021-11-19T08:39:20.513380+0000","last_in_change":"2021-11-19T08:39:05.162873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-19T08:38:42.679405+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-19T08:40:19.691661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-19T08:40:20.621494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"55d41e72-4389-4078-ab49-915578ae7162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6803","nonce":2382765305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6805","nonce":2382765305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6809","nonce":2382765305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6807","nonce":2382765305}]},"public_addr":"172.21.15.23:6803/2382765305","cluster_addr":"172.21.15.23:6805/2382765305","heartbeat_back_addr":"172.21.15.23:6809/2382765305","heartbeat_front_addr":"172.21.15.23:6807/2382765305","state":["exists","up"]},{"osd":1,"uuid":"5b7aea87-24b7-43e6-9ade-8fef0e9e6dd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6811","nonce":1051412153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6813","nonce":1051412153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6817","nonce":1051412153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6815","nonce":1051412153}]},"public_addr":"172.21.15.23:6811/1051412153","cluster_addr":"172.21.15.23:6813/1051412153","heartbeat_back_addr":"172.21.15.23:6817/1051412153","heartbeat_front_addr":"172.21.15.23:6815/1051412153","state":["exists","up"]},{"osd":2,"uuid":"e21cf6ec-4aa2-43cc-934a-bbd8d6e6694c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6819","nonce":954281054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6821","nonce":954281054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6825","nonce":954281054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6823","nonce":954281054}]},"public_addr":"172.21.15.23:6819/954281054","cluster_addr":"172.21.15.23:6821/954281054","heartbeat_back_addr":"172.21.15.23:6825/954281054","heartbeat_front_addr":"172.21.15.23:6823/954281054","state":["exists","up"]},{"osd":3,"uuid":"276cb410-5d23-4cdf-a090-e777ebe4c82f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6801","nonce":2145743581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6803","nonce":2145743581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6807","nonce":2145743581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6805","nonce":2145743581}]},"public_addr":"172.21.15.134:6801/2145743581","cluster_addr":"172.21.15.134:6803/2145743581","heartbeat_back_addr":"172.21.15.134:6807/2145743581","heartbeat_front_addr":"172.21.15.134:6805/2145743581","state":["exists","up"]},{"osd":4,"uuid":"015b6da1-35ed-43f4-84c9-337d2b0e79bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6809","nonce":1867765886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6811","nonce":1867765886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6815","nonce":1867765886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6813","nonce":1867765886}]},"public_addr":"172.21.15.134:6809/1867765886","cluster_addr":"172.21.15.134:6811/1867765886","heartbeat_back_addr":"172.21.15.134:6815/1867765886","heartbeat_front_addr":"172.21.15.134:6813/1867765886","state":["exists","up"]},{"osd":5,"uuid":"86ec41d3-38df-4ee0-a320-402055f7895b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6817","nonce":3784199400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6819","nonce":3784199400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6823","nonce":3784199400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6821","nonce":3784199400}]},"public_addr":"172.21.15.134:6817/3784199400","cluster_addr":"172.21.15.134:6819/3784199400","heartbeat_back_addr":"172.21.15.134:6823/3784199400","heartbeat_front_addr":"172.21.15.134:6821/3784199400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:37:52.252795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:12.408238+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:31.280017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:47.181003+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:39:03.121900+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:39:19.383582+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/192042719":"2021-11-20T08:36:18.730761+0000","172.21.15.23:0/4146945986":"2021-11-20T08:36:18.730761+0000","172.21.15.23:6801/158654977":"2021-11-20T08:36:18.730761+0000","172.21.15.23:6800/158654977":"2021-11-20T08:36:18.730761+0000","172.21.15.23:0/992169182":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/1001097146":"2021-11-20T08:36:02.243722+0000","172.21.15.23:6800/1352671878":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/3039325239":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6801/1352671878":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/4131282839":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6800/945698991":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6801/945698991":"2021-11-20T08:35:42.544921+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-19T08:40:37.814 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.828+0000 7f5030b7b700 1 -- 172.21.15.23:0/2568393709 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f501805c300 msgr2=0x7f501805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:37.814 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.828+0000 7f5030b7b700 1 --2- 172.21.15.23:0/2568393709 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f501805c300 0x7f501805e7c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f501c009f50 tx=0x7f501c009b30).stop 2021-11-19T08:40:37.814 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.828+0000 7f5030b7b700 1 -- 172.21.15.23:0/2568393709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f502c0603b0 msgr2=0x7f502c106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:37.814 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.828+0000 7f5030b7b700 1 --2- 172.21.15.23:0/2568393709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f502c0603b0 0x7f502c106dd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f50140026d0 tx=0x7f5014009de0).stop 2021-11-19T08:40:37.815 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.828+0000 7f5030b7b700 1 -- 172.21.15.23:0/2568393709 shutdown_connections 2021-11-19T08:40:37.815 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.828+0000 7f5030b7b700 1 --2- 172.21.15.23:0/2568393709 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f501805c300 0x7f501805e7c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:37.815 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.828+0000 7f5030b7b700 1 --2- 172.21.15.23:0/2568393709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f502c0603b0 0x7f502c106dd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:37.815 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.828+0000 7f5030b7b700 1 --2- 172.21.15.23:0/2568393709 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f502c0f9760 0x7f502c1072c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:37.816 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.828+0000 7f5030b7b700 1 -- 172.21.15.23:0/2568393709 >> 172.21.15.23:0/2568393709 conn(0x7f502c0f0c30 msgr2=0x7f502c0f3020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:37.816 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.829+0000 7f5030b7b700 1 -- 172.21.15.23:0/2568393709 shutdown_connections 2021-11-19T08:40:37.816 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.829+0000 7f5030b7b700 1 -- 172.21.15.23:0/2568393709 wait complete. 2021-11-19T08:40:37.832 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-19T08:38:42.679405+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-19T08:40:19.691661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-19T08:40:20.621494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-19T08:40:37.833 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-19T08:40:37.961 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.974+0000 7f0e268ac700 1 -- 172.21.15.23:0/3665481378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e200f6820 msgr2=0x7f0e200f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:37.962 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.974+0000 7f0e268ac700 1 --2- 172.21.15.23:0/3665481378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e200f6820 0x7f0e200f6c80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f0e14004660 tx=0x7f0e14009b30).stop 2021-11-19T08:40:37.962 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.974+0000 7f0e268ac700 1 -- 172.21.15.23:0/3665481378 shutdown_connections 2021-11-19T08:40:37.962 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.974+0000 7f0e268ac700 1 --2- 172.21.15.23:0/3665481378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e200f6820 0x7f0e200f6c80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:37.963 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.974+0000 7f0e268ac700 1 --2- 172.21.15.23:0/3665481378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0e200f5580 0x7f0e200f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:37.963 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.974+0000 7f0e268ac700 1 -- 172.21.15.23:0/3665481378 >> 172.21.15.23:0/3665481378 conn(0x7f0e200f0ce0 msgr2=0x7f0e200f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:37.963 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.975+0000 7f0e268ac700 1 -- 172.21.15.23:0/3665481378 shutdown_connections 2021-11-19T08:40:37.963 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.975+0000 7f0e268ac700 1 -- 172.21.15.23:0/3665481378 wait complete. 2021-11-19T08:40:37.963 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.975+0000 7f0e268ac700 1 Processor -- start 2021-11-19T08:40:37.964 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.975+0000 7f0e268ac700 1 -- start start 2021-11-19T08:40:37.964 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.975+0000 7f0e268ac700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e200f5580 0x7f0e2005ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:37.964 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e268ac700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0e200f6820 0x7f0e2010bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:37.964 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e268ac700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2010c100 con 0x7f0e200f5580 2021-11-19T08:40:37.965 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e268ac700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2010c240 con 0x7f0e200f6820 2021-11-19T08:40:37.965 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e1ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e200f5580 0x7f0e2005ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:37.965 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e1ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e200f5580 0x7f0e2005ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47012/0 (socket says 172.21.15.23:47012) 2021-11-19T08:40:37.965 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e1ffff700 1 -- 172.21.15.23:0/2890959173 learned_addr learned my addr 172.21.15.23:0/2890959173 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:37.966 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e1f7fe700 1 --2- 172.21.15.23:0/2890959173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0e200f6820 0x7f0e2010bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:37.966 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e1ffff700 1 -- 172.21.15.23:0/2890959173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0e200f6820 msgr2=0x7f0e2010bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:37.966 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e1ffff700 1 --2- 172.21.15.23:0/2890959173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0e200f6820 0x7f0e2010bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:37.966 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e1ffff700 1 -- 172.21.15.23:0/2890959173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e14005040 con 0x7f0e200f5580 2021-11-19T08:40:37.966 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e1f7fe700 1 --2- 172.21.15.23:0/2890959173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0e200f6820 0x7f0e2010bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:40:37.967 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e1ffff700 1 --2- 172.21.15.23:0/2890959173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e200f5580 0x7f0e2005ff20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f0e1000e7f0 tx=0x7f0e1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:37.967 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e1cff9700 1 -- 172.21.15.23:0/2890959173 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e100092e0 con 0x7f0e200f5580 2021-11-19T08:40:37.967 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e268ac700 1 -- 172.21.15.23:0/2890959173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2010c4a0 con 0x7f0e200f5580 2021-11-19T08:40:37.967 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e1cff9700 1 -- 172.21.15.23:0/2890959173 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e1000f040 con 0x7f0e200f5580 2021-11-19T08:40:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e1cff9700 1 -- 172.21.15.23:0/2890959173 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e100146c0 con 0x7f0e200f5580 2021-11-19T08:40:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.976+0000 7f0e268ac700 1 -- 172.21.15.23:0/2890959173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2010ca60 con 0x7f0e200f5580 2021-11-19T08:40:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.977+0000 7f0e268ac700 1 -- 172.21.15.23:0/2890959173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e20027a00 con 0x7f0e200f5580 2021-11-19T08:40:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.978+0000 7f0e1cff9700 1 -- 172.21.15.23:0/2890959173 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0e1001b030 con 0x7f0e200f5580 2021-11-19T08:40:37.970 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.978+0000 7f0e1cff9700 1 --2- 172.21.15.23:0/2890959173 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0e0805c2a0 0x7f0e0805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:37.970 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.978+0000 7f0e1cff9700 1 -- 172.21.15.23:0/2890959173 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f0e10075fc0 con 0x7f0e200f5580 2021-11-19T08:40:37.970 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.978+0000 7f0e1f7fe700 1 --2- 172.21.15.23:0/2890959173 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0e0805c2a0 0x7f0e0805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:37.970 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.979+0000 7f0e1f7fe700 1 --2- 172.21.15.23:0/2890959173 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0e0805c2a0 0x7f0e0805e760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f0e14005010 tx=0x7f0e1400b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:37.971 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:37.982+0000 7f0e1cff9700 1 -- 172.21.15.23:0/2890959173 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e100a22b0 con 0x7f0e200f5580 2021-11-19T08:40:38.135 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.149+0000 7f0e268ac700 1 -- 172.21.15.23:0/2890959173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f0e200fdd10 con 0x7f0e200f5580 2021-11-19T08:40:38.135 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.150+0000 7f0e1cff9700 1 -- 172.21.15.23:0/2890959173 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v41) v1 ==== 110+0+10 (secure 0 0 0) 0x7f0e10019020 con 0x7f0e200f5580 2021-11-19T08:40:38.135 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2021-11-19T08:40:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.151+0000 7f0e268ac700 1 -- 172.21.15.23:0/2890959173 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0e0805c2a0 msgr2=0x7f0e0805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.151+0000 7f0e268ac700 1 --2- 172.21.15.23:0/2890959173 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0e0805c2a0 0x7f0e0805e760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f0e14005010 tx=0x7f0e1400b040).stop 2021-11-19T08:40:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.151+0000 7f0e268ac700 1 -- 172.21.15.23:0/2890959173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e200f5580 msgr2=0x7f0e2005ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:38.137 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.151+0000 7f0e268ac700 1 --2- 172.21.15.23:0/2890959173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e200f5580 0x7f0e2005ff20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f0e1000e7f0 tx=0x7f0e1000b040).stop 2021-11-19T08:40:38.138 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.152+0000 7f0e268ac700 1 -- 172.21.15.23:0/2890959173 shutdown_connections 2021-11-19T08:40:38.138 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.152+0000 7f0e268ac700 1 --2- 172.21.15.23:0/2890959173 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0e0805c2a0 0x7f0e0805e760 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.138 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.152+0000 7f0e268ac700 1 --2- 172.21.15.23:0/2890959173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e200f5580 0x7f0e2005ff20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.138 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.152+0000 7f0e268ac700 1 --2- 172.21.15.23:0/2890959173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0e200f6820 0x7f0e2010bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.139 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.152+0000 7f0e268ac700 1 -- 172.21.15.23:0/2890959173 >> 172.21.15.23:0/2890959173 conn(0x7f0e200f0ce0 msgr2=0x7f0e200fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:38.139 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.152+0000 7f0e268ac700 1 -- 172.21.15.23:0/2890959173 shutdown_connections 2021-11-19T08:40:38.139 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.152+0000 7f0e268ac700 1 -- 172.21.15.23:0/2890959173 wait complete. 2021-11-19T08:40:38.149 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-19T08:40:38.286 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.297+0000 7f13e3266700 1 -- 172.21.15.23:0/3675128661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13dc0f6820 msgr2=0x7f13dc0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:38.286 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.297+0000 7f13e3266700 1 --2- 172.21.15.23:0/3675128661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13dc0f6820 0x7f13dc0f6c80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f13cc004660 tx=0x7f13cc009b30).stop 2021-11-19T08:40:38.287 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.297+0000 7f13e3266700 1 -- 172.21.15.23:0/3675128661 shutdown_connections 2021-11-19T08:40:38.287 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.297+0000 7f13e3266700 1 --2- 172.21.15.23:0/3675128661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13dc0f6820 0x7f13dc0f6c80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.287 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.297+0000 7f13e3266700 1 --2- 172.21.15.23:0/3675128661 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13dc0f5580 0x7f13dc0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.287 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.297+0000 7f13e3266700 1 -- 172.21.15.23:0/3675128661 >> 172.21.15.23:0/3675128661 conn(0x7f13dc0f0ce0 msgr2=0x7f13dc0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:38.288 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.297+0000 7f13e3266700 1 -- 172.21.15.23:0/3675128661 shutdown_connections 2021-11-19T08:40:38.288 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.297+0000 7f13e3266700 1 -- 172.21.15.23:0/3675128661 wait complete. 2021-11-19T08:40:38.288 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13e3266700 1 Processor -- start 2021-11-19T08:40:38.288 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13e3266700 1 -- start start 2021-11-19T08:40:38.288 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13e3266700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13dc0f5580 0x7f13dc05ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13e3266700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13dc0f6820 0x7f13dc10bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13e3266700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc10c120 con 0x7f13dc0f6820 2021-11-19T08:40:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13e3266700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc10c260 con 0x7f13dc0f5580 2021-11-19T08:40:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13e0801700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13dc0f6820 0x7f13dc10bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13e0801700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13dc0f6820 0x7f13dc10bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47024/0 (socket says 172.21.15.23:47024) 2021-11-19T08:40:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13e0801700 1 -- 172.21.15.23:0/4109040086 learned_addr learned my addr 172.21.15.23:0/4109040086 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13e1002700 1 --2- 172.21.15.23:0/4109040086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13dc0f5580 0x7f13dc05ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13e0801700 1 -- 172.21.15.23:0/4109040086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13dc0f5580 msgr2=0x7f13dc05ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13e0801700 1 --2- 172.21.15.23:0/4109040086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13dc0f5580 0x7f13dc05ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13e0801700 1 -- 172.21.15.23:0/4109040086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13cc005040 con 0x7f13dc0f6820 2021-11-19T08:40:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13e0801700 1 --2- 172.21.15.23:0/4109040086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13dc0f6820 0x7f13dc10bc30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f13cc000c00 tx=0x7f13cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.298+0000 7f13d1ffb700 1 -- 172.21.15.23:0/4109040086 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13cc019070 con 0x7f13dc0f6820 2021-11-19T08:40:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.299+0000 7f13d1ffb700 1 -- 172.21.15.23:0/4109040086 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13cc007e70 con 0x7f13dc0f6820 2021-11-19T08:40:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.299+0000 7f13d1ffb700 1 -- 172.21.15.23:0/4109040086 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13cc014c90 con 0x7f13dc0f6820 2021-11-19T08:40:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.299+0000 7f13e3266700 1 -- 172.21.15.23:0/4109040086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13dc10c460 con 0x7f13dc0f6820 2021-11-19T08:40:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.299+0000 7f13e3266700 1 -- 172.21.15.23:0/4109040086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13dc10c990 con 0x7f13dc0f6820 2021-11-19T08:40:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.300+0000 7f13e3266700 1 -- 172.21.15.23:0/4109040086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13dc046330 con 0x7f13dc0f6820 2021-11-19T08:40:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.300+0000 7f13d1ffb700 1 -- 172.21.15.23:0/4109040086 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f13cc025070 con 0x7f13dc0f6820 2021-11-19T08:40:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.300+0000 7f13d1ffb700 1 --2- 172.21.15.23:0/4109040086 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f13c805c2a0 0x7f13c805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.300+0000 7f13d1ffb700 1 -- 172.21.15.23:0/4109040086 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f13cc076e10 con 0x7f13dc0f6820 2021-11-19T08:40:38.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.300+0000 7f13e1002700 1 --2- 172.21.15.23:0/4109040086 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f13c805c2a0 0x7f13c805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:38.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.301+0000 7f13e1002700 1 --2- 172.21.15.23:0/4109040086 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f13c805c2a0 0x7f13c805e760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f13d8009fd0 tx=0x7f13d800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:38.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.305+0000 7f13d1ffb700 1 -- 172.21.15.23:0/4109040086 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13cc049b50 con 0x7f13dc0f6820 2021-11-19T08:40:38.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.467+0000 7f13e3266700 1 -- 172.21.15.23:0/4109040086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f13dc027a00 con 0x7f13dc0f6820 2021-11-19T08:40:38.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.467+0000 7f13d1ffb700 1 -- 172.21.15.23:0/4109040086 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7f13cc04d170 con 0x7f13dc0f6820 2021-11-19T08:40:38.453 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2021-11-19T08:40:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.469+0000 7f13e3266700 1 -- 172.21.15.23:0/4109040086 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f13c805c2a0 msgr2=0x7f13c805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:38.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.469+0000 7f13e3266700 1 --2- 172.21.15.23:0/4109040086 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f13c805c2a0 0x7f13c805e760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f13d8009fd0 tx=0x7f13d800b040).stop 2021-11-19T08:40:38.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.469+0000 7f13e3266700 1 -- 172.21.15.23:0/4109040086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13dc0f6820 msgr2=0x7f13dc10bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:38.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.469+0000 7f13e3266700 1 --2- 172.21.15.23:0/4109040086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13dc0f6820 0x7f13dc10bc30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f13cc000c00 tx=0x7f13cc00b040).stop 2021-11-19T08:40:38.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.469+0000 7f13e3266700 1 -- 172.21.15.23:0/4109040086 shutdown_connections 2021-11-19T08:40:38.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.469+0000 7f13e3266700 1 --2- 172.21.15.23:0/4109040086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13dc0f5580 0x7f13dc05ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.469+0000 7f13e3266700 1 --2- 172.21.15.23:0/4109040086 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f13c805c2a0 0x7f13c805e760 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.469+0000 7f13e3266700 1 --2- 172.21.15.23:0/4109040086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13dc0f6820 0x7f13dc10bc30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.469+0000 7f13e3266700 1 -- 172.21.15.23:0/4109040086 >> 172.21.15.23:0/4109040086 conn(0x7f13dc0f0ce0 msgr2=0x7f13dc0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:38.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.469+0000 7f13e3266700 1 -- 172.21.15.23:0/4109040086 shutdown_connections 2021-11-19T08:40:38.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.469+0000 7f13e3266700 1 -- 172.21.15.23:0/4109040086 wait complete. 2021-11-19T08:40:38.465 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:38 smithi023 conmon[33490]: cluster 2021-11-19T08:40:36.828368+0000 mgr.smithi023.ehlpcl (mgr.14164) 198 : cluster [DBG] pgmap v159: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2021-11-19T08:40:38.465 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:38 smithi023 conmon[33490]: cluster 2021-11-19T08:40:37.181689+0000 mon.smithi023 (mon.0) 525 : cluster [DBG] mds.? [v2:172.21.15.23:6828/2176063367,v1:172.21.15.23:6829/2176063367] up:standby-replay 2021-11-19T08:40:38.466 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:38 smithi023 conmon[33490]: cluster 2021-11-19T08:40:37.181744+0000 mon.smithi023 (mon.0) 526 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi023.fuvzkr=up:active,1=cephfs.smithi134.opxllb=up:active} 2 up:standby-replay 2021-11-19T08:40:38.466 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:38 smithi023 conmon[33490]: audit 2021-11-19T08:40:37.236742+0000 mon.smithi023 (mon.0) 527 : audit [INF] 2021-11-19T08:40:38.466 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:38 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:38.466 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:38 smithi023 conmon[33490]: audit 2021-11-19T08:40:37.240131+0000 mon.smithi023 (mon.0) 528 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:38.467 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:38 smithi023 conmon[33490]: audit 2021-11-19T08:40:37.243773+0000 mon.smithi023 (mon.0) 529 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:40:38.467 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:38 smithi023 conmon[33490]: audit 2021-11-19T08:40:37.827055+0000 mon.smithi023 (mon.0) 530 : audit [DBG] from='client.? 172.21.15.23:0/2568393709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-19T08:40:38.467 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:38 smithi023 conmon[33490]: audit 2021-11-19T08:40:38.150981+0000 mon.smithi023 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.23:0/2890959173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-19T08:40:38.468 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-19T08:40:38.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:38 smithi134 conmon[36363]: cluster 2021-11-19T08:40:36.828368+0000 mgr.smithi023.ehlpcl (mgr.14164) 198 : cluster 2021-11-19T08:40:38.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:38 smithi134 conmon[36363]: [DBG] pgmap v159: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2021-11-19T08:40:38.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:38 smithi134 conmon[36363]: cluster 2021-11-19T08:40:37.181689+0000 mon.smithi023 (mon.0) 525 : cluster [DBG] mds.? [v2:172.21.15.23:6828/2176063367,v1:172.21.15.23:6829/2176063367] up:standby-replay 2021-11-19T08:40:38.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:38 smithi134 conmon[36363]: cluster 2021-11-19T08:40:37.181744+0000 mon.smithi023 (mon.0) 526 : cluster [DBG] fsmap cephfs:2 {0=cephfs.smithi023.fuvzkr=up:active,1=cephfs.smithi134.opxllb=up:active} 2 up:standby-replay 2021-11-19T08:40:38.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:38 smithi134 conmon[36363]: audit 2021-11-19T08:40:37.236742+0000 mon.smithi023 (mon.0) 527 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:38.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:38 smithi134 conmon[36363]: audit 2021-11-19T08:40:37.240131+0000 mon.smithi023 (mon.0) 528 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:38.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:38 smithi134 conmon[36363]: audit 2021-11-19T08:40:37.243773+0000 mon.smithi023 (mon.0) 529 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:40:38.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:38 smithi134 conmon[36363]: audit 2021-11-19T08:40:37.827055+0000 mon.smithi023 (mon.0) 530 : audit [DBG] from='client.? 172.21.15.23:0/2568393709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-19T08:40:38.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:38 smithi134 conmon[36363]: audit 2021-11-19T08:40:38.150981+0000 mon.smithi023 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.23:0/2890959173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-19T08:40:38.600 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.612+0000 7f214f977700 1 -- 172.21.15.23:0/1968129759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21480f7b40 msgr2=0x7f21480f7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:38.601 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.612+0000 7f214f977700 1 --2- 172.21.15.23:0/1968129759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21480f7b40 0x7f21480f7fa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f213c004660 tx=0x7f213c009b30).stop 2021-11-19T08:40:38.601 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.612+0000 7f214f977700 1 -- 172.21.15.23:0/1968129759 shutdown_connections 2021-11-19T08:40:38.601 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.612+0000 7f214f977700 1 --2- 172.21.15.23:0/1968129759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21480f7b40 0x7f21480f7fa0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.601 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.612+0000 7f214f977700 1 --2- 172.21.15.23:0/1968129759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21480f68a0 0x7f21480f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.602 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.612+0000 7f214f977700 1 -- 172.21.15.23:0/1968129759 >> 172.21.15.23:0/1968129759 conn(0x7f21480f1fc0 msgr2=0x7f21480f4420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:38.602 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.612+0000 7f214f977700 1 -- 172.21.15.23:0/1968129759 shutdown_connections 2021-11-19T08:40:38.602 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.612+0000 7f214f977700 1 -- 172.21.15.23:0/1968129759 wait complete. 2021-11-19T08:40:38.602 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.613+0000 7f214f977700 1 Processor -- start 2021-11-19T08:40:38.603 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.613+0000 7f214f977700 1 -- start start 2021-11-19T08:40:38.603 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.613+0000 7f214f977700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21480f68a0 0x7f214806b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:38.603 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.613+0000 7f214f977700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21480f7b40 0x7f2148069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:38.603 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.613+0000 7f214f977700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214806bd70 con 0x7f21480f68a0 2021-11-19T08:40:38.603 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.613+0000 7f214f977700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214806a310 con 0x7f21480f7b40 2021-11-19T08:40:38.604 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.613+0000 7f214d713700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21480f68a0 0x7f214806b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:38.604 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.613+0000 7f214d713700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21480f68a0 0x7f214806b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47030/0 (socket says 172.21.15.23:47030) 2021-11-19T08:40:38.604 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.613+0000 7f214d713700 1 -- 172.21.15.23:0/771310753 learned_addr learned my addr 172.21.15.23:0/771310753 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:38.604 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.613+0000 7f214cf12700 1 --2- 172.21.15.23:0/771310753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21480f7b40 0x7f2148069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:38.605 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.613+0000 7f214d713700 1 -- 172.21.15.23:0/771310753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21480f7b40 msgr2=0x7f2148069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:38.605 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.613+0000 7f214d713700 1 --2- 172.21.15.23:0/771310753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21480f7b40 0x7f2148069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.605 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.613+0000 7f214d713700 1 -- 172.21.15.23:0/771310753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213c005040 con 0x7f21480f68a0 2021-11-19T08:40:38.605 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.613+0000 7f214cf12700 1 --2- 172.21.15.23:0/771310753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21480f7b40 0x7f2148069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:40:38.605 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.614+0000 7f214d713700 1 --2- 172.21.15.23:0/771310753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21480f68a0 0x7f214806b7c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f2144002700 tx=0x7f2144009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:38.606 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.614+0000 7f2139ffb700 1 -- 172.21.15.23:0/771310753 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2144009320 con 0x7f21480f68a0 2021-11-19T08:40:38.606 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.614+0000 7f214f977700 1 -- 172.21.15.23:0/771310753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214806a590 con 0x7f21480f68a0 2021-11-19T08:40:38.606 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.614+0000 7f2139ffb700 1 -- 172.21.15.23:0/771310753 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f214400f040 con 0x7f21480f68a0 2021-11-19T08:40:38.606 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.614+0000 7f2139ffb700 1 -- 172.21.15.23:0/771310753 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2144014900 con 0x7f21480f68a0 2021-11-19T08:40:38.607 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.614+0000 7f214f977700 1 -- 172.21.15.23:0/771310753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214806ab50 con 0x7f21480f68a0 2021-11-19T08:40:38.607 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.615+0000 7f214f977700 1 -- 172.21.15.23:0/771310753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f214805ea10 con 0x7f21480f68a0 2021-11-19T08:40:38.607 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.615+0000 7f2139ffb700 1 -- 172.21.15.23:0/771310753 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f21440077b0 con 0x7f21480f68a0 2021-11-19T08:40:38.607 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.615+0000 7f2139ffb700 1 --2- 172.21.15.23:0/771310753 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f213405c2a0 0x7f213405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:38.608 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.615+0000 7f2139ffb700 1 -- 172.21.15.23:0/771310753 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f2144076200 con 0x7f21480f68a0 2021-11-19T08:40:38.608 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.615+0000 7f214cf12700 1 --2- 172.21.15.23:0/771310753 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f213405c2a0 0x7f213405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:38.608 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.616+0000 7f214cf12700 1 --2- 172.21.15.23:0/771310753 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f213405c2a0 0x7f213405e760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f213c005010 tx=0x7f213c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:38.608 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.619+0000 7f2139ffb700 1 -- 172.21.15.23:0/771310753 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2144048ec0 con 0x7f21480f68a0 2021-11-19T08:40:38.765 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.780+0000 7f214f977700 1 -- 172.21.15.23:0/771310753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f21480ff100 con 0x7f21480f68a0 2021-11-19T08:40:38.766 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.780+0000 7f2139ffb700 1 -- 172.21.15.23:0/771310753 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7f214404c4e0 con 0x7f21480f68a0 2021-11-19T08:40:38.766 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2021-11-19T08:40:38.767 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.782+0000 7f214f977700 1 -- 172.21.15.23:0/771310753 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f213405c2a0 msgr2=0x7f213405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:38.768 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.782+0000 7f214f977700 1 --2- 172.21.15.23:0/771310753 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f213405c2a0 0x7f213405e760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f213c005010 tx=0x7f213c009b30).stop 2021-11-19T08:40:38.768 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.782+0000 7f214f977700 1 -- 172.21.15.23:0/771310753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21480f68a0 msgr2=0x7f214806b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:38.768 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.782+0000 7f214f977700 1 --2- 172.21.15.23:0/771310753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21480f68a0 0x7f214806b7c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f2144002700 tx=0x7f2144009d80).stop 2021-11-19T08:40:38.768 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.782+0000 7f214f977700 1 -- 172.21.15.23:0/771310753 shutdown_connections 2021-11-19T08:40:38.769 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.782+0000 7f214f977700 1 --2- 172.21.15.23:0/771310753 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f213405c2a0 0x7f213405e760 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.769 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.782+0000 7f214f977700 1 --2- 172.21.15.23:0/771310753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21480f68a0 0x7f214806b7c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.769 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.782+0000 7f214f977700 1 --2- 172.21.15.23:0/771310753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21480f7b40 0x7f2148069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.769 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.782+0000 7f214f977700 1 -- 172.21.15.23:0/771310753 >> 172.21.15.23:0/771310753 conn(0x7f21480f1fc0 msgr2=0x7f21480fb590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:38.769 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.783+0000 7f214f977700 1 -- 172.21.15.23:0/771310753 shutdown_connections 2021-11-19T08:40:38.770 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.783+0000 7f214f977700 1 -- 172.21.15.23:0/771310753 wait complete. 2021-11-19T08:40:38.780 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:40:38.908 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.921+0000 7ffa2759e700 1 -- 172.21.15.23:0/3872312195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa280f6770 msgr2=0x7ffa280f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:38.909 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.921+0000 7ffa2759e700 1 --2- 172.21.15.23:0/3872312195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa280f6770 0x7ffa280f6bd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7ffa1c004660 tx=0x7ffa1c009b30).stop 2021-11-19T08:40:38.909 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.921+0000 7ffa2759e700 1 -- 172.21.15.23:0/3872312195 shutdown_connections 2021-11-19T08:40:38.909 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.921+0000 7ffa2759e700 1 --2- 172.21.15.23:0/3872312195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa280f6770 0x7ffa280f6bd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.909 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.921+0000 7ffa2759e700 1 --2- 172.21.15.23:0/3872312195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa280f54d0 0x7ffa280f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.910 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.921+0000 7ffa2759e700 1 -- 172.21.15.23:0/3872312195 >> 172.21.15.23:0/3872312195 conn(0x7ffa280f0c10 msgr2=0x7ffa280f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:38.910 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.921+0000 7ffa2759e700 1 -- 172.21.15.23:0/3872312195 shutdown_connections 2021-11-19T08:40:38.910 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.921+0000 7ffa2759e700 1 -- 172.21.15.23:0/3872312195 wait complete. 2021-11-19T08:40:38.910 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.922+0000 7ffa2759e700 1 Processor -- start 2021-11-19T08:40:38.910 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.922+0000 7ffa2759e700 1 -- start start 2021-11-19T08:40:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.922+0000 7ffa2759e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa280f54d0 0x7ffa281099c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.922+0000 7ffa2759e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa280f6770 0x7ffa2810bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.922+0000 7ffa2759e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa280605a0 con 0x7ffa280f6770 2021-11-19T08:40:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.922+0000 7ffa2759e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa280606e0 con 0x7ffa280f54d0 2021-11-19T08:40:38.912 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.922+0000 7ffa25d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa280f6770 0x7ffa2810bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:38.912 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.922+0000 7ffa25d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa280f6770 0x7ffa2810bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47040/0 (socket says 172.21.15.23:47040) 2021-11-19T08:40:38.912 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.922+0000 7ffa25d9b700 1 -- 172.21.15.23:0/3025056070 learned_addr learned my addr 172.21.15.23:0/3025056070 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:38.912 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.922+0000 7ffa2659c700 1 --2- 172.21.15.23:0/3025056070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa280f54d0 0x7ffa281099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:38.912 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.922+0000 7ffa25d9b700 1 -- 172.21.15.23:0/3025056070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa280f54d0 msgr2=0x7ffa281099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:38.913 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.922+0000 7ffa25d9b700 1 --2- 172.21.15.23:0/3025056070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa280f54d0 0x7ffa281099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:38.913 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.922+0000 7ffa25d9b700 1 -- 172.21.15.23:0/3025056070 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c005040 con 0x7ffa280f6770 2021-11-19T08:40:38.914 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.923+0000 7ffa25d9b700 1 --2- 172.21.15.23:0/3025056070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa280f6770 0x7ffa2810bec0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7ffa1c009f50 tx=0x7ffa1c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:38.914 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.923+0000 7ffa16ffd700 1 -- 172.21.15.23:0/3025056070 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa1c018070 con 0x7ffa280f6770 2021-11-19T08:40:38.914 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.923+0000 7ffa16ffd700 1 -- 172.21.15.23:0/3025056070 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa1c013410 con 0x7ffa280f6770 2021-11-19T08:40:38.914 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.923+0000 7ffa2759e700 1 -- 172.21.15.23:0/3025056070 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2810c410 con 0x7ffa280f6770 2021-11-19T08:40:38.915 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.923+0000 7ffa16ffd700 1 -- 172.21.15.23:0/3025056070 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa1c01c3c0 con 0x7ffa280f6770 2021-11-19T08:40:38.915 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.923+0000 7ffa2759e700 1 -- 172.21.15.23:0/3025056070 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2810c970 con 0x7ffa280f6770 2021-11-19T08:40:38.915 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.924+0000 7ffa2759e700 1 -- 172.21.15.23:0/3025056070 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa28045ad0 con 0x7ffa280f6770 2021-11-19T08:40:38.917 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.924+0000 7ffa16ffd700 1 -- 172.21.15.23:0/3025056070 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ffa1c01d420 con 0x7ffa280f6770 2021-11-19T08:40:38.917 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.924+0000 7ffa16ffd700 1 --2- 172.21.15.23:0/3025056070 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ffa1005c300 0x7ffa1005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:38.917 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.924+0000 7ffa16ffd700 1 -- 172.21.15.23:0/3025056070 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7ffa1c077b40 con 0x7ffa280f6770 2021-11-19T08:40:38.918 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.924+0000 7ffa2659c700 1 --2- 172.21.15.23:0/3025056070 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ffa1005c300 0x7ffa1005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:38.918 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.925+0000 7ffa2659c700 1 --2- 172.21.15.23:0/3025056070 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ffa1005c300 0x7ffa1005e7c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7ffa28060220 tx=0x7ffa1800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:38.918 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:38.928+0000 7ffa16ffd700 1 -- 172.21.15.23:0/3025056070 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa1c04a800 con 0x7ffa280f6770 2021-11-19T08:40:39.115 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:39.130+0000 7ffa2759e700 1 -- 172.21.15.23:0/3025056070 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa28027a00 con 0x7ffa280f6770 2021-11-19T08:40:39.116 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:39.131+0000 7ffa16ffd700 1 -- 172.21.15.23:0/3025056070 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2962 (secure 0 0 0) 0x7ffa1c04de20 con 0x7ffa280f6770 2021-11-19T08:40:39.116 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:39.116 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":14,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:40:37.178115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14434,"mds_1":24283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14434":{"gid":14434,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.23:6827/530484646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":530484646},{"type":"v1","addr":"172.21.15.23:6827","nonce":530484646}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14446":{"gid":14446,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.23:6829/2176063367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2176063367},{"type":"v1","addr":"172.21.15.23:6829","nonce":2176063367}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24271":{"gid":24271,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.134:6825/1874302373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1874302373},{"type":"v1","addr":"172.21.15.134:6825","nonce":1874302373}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24283":{"gid":24283,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":8,"state":"up:active","state_seq":2,"addr":"172.21.15.134:6827/3905671987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3905671987},{"type":"v1","addr":"172.21.15.134:6827","nonce":3905671987}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:40:39.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:39.132+0000 7ffa2759e700 1 -- 172.21.15.23:0/3025056070 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ffa1005c300 msgr2=0x7ffa1005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:39.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:39.132+0000 7ffa2759e700 1 --2- 172.21.15.23:0/3025056070 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ffa1005c300 0x7ffa1005e7c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7ffa28060220 tx=0x7ffa1800b040).stop 2021-11-19T08:40:39.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:39.132+0000 7ffa2759e700 1 -- 172.21.15.23:0/3025056070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa280f6770 msgr2=0x7ffa2810bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:39.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:39.132+0000 7ffa2759e700 1 --2- 172.21.15.23:0/3025056070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa280f6770 0x7ffa2810bec0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7ffa1c009f50 tx=0x7ffa1c009b30).stop 2021-11-19T08:40:39.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:39.132+0000 7ffa2759e700 1 -- 172.21.15.23:0/3025056070 shutdown_connections 2021-11-19T08:40:39.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:39.132+0000 7ffa2759e700 1 --2- 172.21.15.23:0/3025056070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa280f54d0 0x7ffa281099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:39.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:39.132+0000 7ffa2759e700 1 --2- 172.21.15.23:0/3025056070 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ffa1005c300 0x7ffa1005e7c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:39.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:39.132+0000 7ffa2759e700 1 --2- 172.21.15.23:0/3025056070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa280f6770 0x7ffa2810bec0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:39.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:39.132+0000 7ffa2759e700 1 -- 172.21.15.23:0/3025056070 >> 172.21.15.23:0/3025056070 conn(0x7ffa280f0c10 msgr2=0x7ffa280fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:39.120 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:39.132+0000 7ffa2759e700 1 -- 172.21.15.23:0/3025056070 shutdown_connections 2021-11-19T08:40:39.120 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:39.133+0000 7ffa2759e700 1 -- 172.21.15.23:0/3025056070 wait complete. 2021-11-19T08:40:39.120 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 14 2021-11-19T08:40:39.130 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 14, 'max_mds': 2, 'flags': 50} 2021-11-19T08:40:39.130 INFO:teuthology.run_tasks:Running task kclient... 2021-11-19T08:40:39.153 INFO:tasks.kclient:Mounting kernel clients... 2021-11-19T08:40:39.154 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2021-11-19T08:40:39.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:39.155 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2021-11-19T08:40:39.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:39.172 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2021-11-19T08:40:39.240 INFO:teuthology.orchestra.run.smithi023.stderr:Cannot find device "ceph-brx" 2021-11-19T08:40:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-19T08:40:39.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:39.242 DEBUG:teuthology.orchestra.run.smithi134:> ip netns list 2021-11-19T08:40:39.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:39.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link delete ceph-brx 2021-11-19T08:40:39.329 INFO:teuthology.orchestra.run.smithi134.stderr:Cannot find device "ceph-brx" 2021-11-19T08:40:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-19T08:40:39.331 INFO:tasks.kclient:client.0 config is {} 2021-11-19T08:40:39.332 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2021-11-19T08:40:39.333 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2021-11-19T08:40:39.333 INFO:tasks.cephfs.mount:self.client.name = client.0 2021-11-19T08:40:39.334 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2021-11-19T08:40:39.335 INFO:tasks.cephfs.mount:self.cephfs_name = None 2021-11-19T08:40:39.335 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2021-11-19T08:40:39.335 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2021-11-19T08:40:39.336 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2021-11-19T08:40:39.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:39.337 DEBUG:teuthology.orchestra.run.smithi023:> ip addr 2021-11-19T08:40:39.355 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2021-11-19T08:40:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2021-11-19T08:40:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2021-11-19T08:40:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2021-11-19T08:40:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2021-11-19T08:40:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2021-11-19T08:40:39.357 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-19T08:40:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2021-11-19T08:40:39.358 INFO:teuthology.orchestra.run.smithi023.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-19T08:40:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2021-11-19T08:40:39.358 INFO:teuthology.orchestra.run.smithi023.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2021-11-19T08:40:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2021-11-19T08:40:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2021-11-19T08:40:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 42261sec preferred_lft 42261sec 2021-11-19T08:40:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link noprefixroute 2021-11-19T08:40:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2021-11-19T08:40:39.360 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-19T08:40:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2021-11-19T08:40:39.361 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2021-11-19T08:40:39.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:39.362 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2021-11-19T08:40:39.363 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2021-11-19T08:40:39.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2021-11-19T08:40:39.365 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2021-11-19T08:40:39.365 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2021-11-19T08:40:39.366 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2021-11-19T08:40:39.366 DEBUG:teuthology.orchestra.run.smithi023:> ') 2021-11-19T08:40:39.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:39 smithi023 conmon[33490]: audit 2021-11-19T08:40:38.468669+0000 mon.smithi023 ( 2021-11-19T08:40:39.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:39 smithi023 conmon[33490]: mon.0) 532 : audit [DBG] from='client.? 172.21.15.23:0/4109040086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-19T08:40:39.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:39 smithi023 conmon[33490]: audit 2021-11-19T08:40:38.781686+0000 mon.smithi023 (mon.0) 533 : audit [DBG] from='client.? 172.21.15.23:0/771310753' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-19T08:40:39.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:39 smithi023 conmon[33490]: audit 2021-11-19T08:40:39.131903+0000 mon.smithi023 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.23:0/3025056070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:40:39.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:39.521 DEBUG:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2021-11-19T08:40:39.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:39 smithi134 conmon[36363]: audit 2021-11-19T08:40:38.468669+0000 mon.smithi023 (mon.0) 532 : audit [DBG] from='client.? 172.21.15.23:0/4109040086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-19T08:40:39.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:39 smithi134 conmon[36363]: audit 2021-11-19T08:40:38.781686+0000 mon.smithi023 (mon.0) 533 : audit [DBG] from='client.? 172.21.15.23:0/771310753' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-19T08:40:39.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:39 smithi134 conmon[36363]: audit 2021-11 2021-11-19T08:40:39.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:39 smithi134 conmon[36363]: -19T08:40:39.131903+0000 mon.smithi023 (mon.0) 534 : audit [DBG] 2021-11-19T08:40:39.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:39 smithi134 conmon[36363]: from='client.? 172.21.15.23:0/3025056070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:40:39.550 INFO:teuthology.orchestra.run.smithi023.stdout:1 2021-11-19T08:40:39.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:39.552 DEBUG:teuthology.orchestra.run.smithi023:> route 2021-11-19T08:40:39.622 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2021-11-19T08:40:39.622 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2021-11-19T08:40:39.623 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2021-11-19T08:40:39.623 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2021-11-19T08:40:39.623 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2021-11-19T08:40:39.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:39.625 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2021-11-19T08:40:39.625 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2021-11-19T08:40:39.626 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2021-11-19T08:40:39.626 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2021-11-19T08:40:39.627 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2021-11-19T08:40:39.627 DEBUG:teuthology.orchestra.run.smithi023:> ') 2021-11-19T08:40:40.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:40.040 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2021-11-19T08:40:40.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:40.058 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list-id 2021-11-19T08:40:40.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:40.115 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2021-11-19T08:40:40.115 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2021-11-19T08:40:40.115 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2021-11-19T08:40:40.116 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2021-11-19T08:40:40.116 DEBUG:teuthology.orchestra.run.smithi023:> ') 2021-11-19T08:40:40.209 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2021-11-19T08:40:40.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:40.210 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2021-11-19T08:40:40.211 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2021-11-19T08:40:40.211 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2021-11-19T08:40:40.212 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2021-11-19T08:40:40.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2021-11-19T08:40:40.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2021-11-19T08:40:40.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2021-11-19T08:40:40.214 DEBUG:teuthology.orchestra.run.smithi023:> ') 2021-11-19T08:40:40.289 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:40 smithi023 conmon[33490]: cluster 2021-11-19T08:40:38.828750+0000 mgr.smithi023.ehlpcl (mgr.14164) 199 : cluster [DBG] pgmap v160: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2021-11-19T08:40:40.289 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:40 smithi023 conmon[33490]: audit 2021-11-19T08:40:40.105799+0000 mon.smithi023 (mon.0) 535 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:40.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:40.457 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2021-11-19T08:40:40.457 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2021-11-19T08:40:40.457 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2021-11-19T08:40:40.458 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2021-11-19T08:40:40.458 DEBUG:teuthology.orchestra.run.smithi023:> ') 2021-11-19T08:40:40.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:40 smithi134 conmon[36363]: cluster 2021-11-19T08:40:38.828750+0000 mgr.smithi023.ehlpcl (mgr.14164) 199 : cluster [DBG] pgmap v160: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2021-11-19T08:40:40.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:40 smithi134 conmon[36363]: audit 2021-11-19T08:40:40.105799+0000 mon.smithi023 (mon.0) 535 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:40.568 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-19T08:40:40.694 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.707+0000 7f4f82b44700 1 -- 172.21.15.23:0/3667506560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7c0f7510 msgr2=0x7f4f7c0f47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:40.694 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.707+0000 7f4f82b44700 1 --2- 172.21.15.23:0/3667506560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7c0f7510 0x7f4f7c0f47a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f4f68004660 tx=0x7f4f68009b30).stop 2021-11-19T08:40:40.694 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.707+0000 7f4f82b44700 1 -- 172.21.15.23:0/3667506560 shutdown_connections 2021-11-19T08:40:40.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.707+0000 7f4f82b44700 1 --2- 172.21.15.23:0/3667506560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7c0f7510 0x7f4f7c0f47a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:40.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.707+0000 7f4f82b44700 1 --2- 172.21.15.23:0/3667506560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f7c0f6b70 0x7f4f7c0f6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:40.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.707+0000 7f4f82b44700 1 -- 172.21.15.23:0/3667506560 >> 172.21.15.23:0/3667506560 conn(0x7f4f7c0f0440 msgr2=0x7f4f7c0f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:40.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.707+0000 7f4f82b44700 1 -- 172.21.15.23:0/3667506560 shutdown_connections 2021-11-19T08:40:40.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.707+0000 7f4f82b44700 1 -- 172.21.15.23:0/3667506560 wait complete. 2021-11-19T08:40:40.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.707+0000 7f4f82b44700 1 Processor -- start 2021-11-19T08:40:40.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.707+0000 7f4f82b44700 1 -- start start 2021-11-19T08:40:40.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.708+0000 7f4f82b44700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f7c0f6b70 0x7f4f7c108fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:40.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.708+0000 7f4f82b44700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7c0f7510 0x7f4f7c1094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:40.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.708+0000 7f4f82b44700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c109a70 con 0x7f4f7c0f7510 2021-11-19T08:40:40.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.708+0000 7f4f82b44700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c109bb0 con 0x7f4f7c0f6b70 2021-11-19T08:40:40.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.708+0000 7f4f7bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7c0f7510 0x7f4f7c1094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:40.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.708+0000 7f4f7bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7c0f7510 0x7f4f7c1094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47050/0 (socket says 172.21.15.23:47050) 2021-11-19T08:40:40.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.708+0000 7f4f7bfff700 1 -- 172.21.15.23:0/3456459583 learned_addr learned my addr 172.21.15.23:0/3456459583 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:40.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.708+0000 7f4f808e0700 1 --2- 172.21.15.23:0/3456459583 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f7c0f6b70 0x7f4f7c108fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:40.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.708+0000 7f4f7bfff700 1 -- 172.21.15.23:0/3456459583 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f7c0f6b70 msgr2=0x7f4f7c108fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:40.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.708+0000 7f4f7bfff700 1 --2- 172.21.15.23:0/3456459583 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f7c0f6b70 0x7f4f7c108fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:40.699 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.708+0000 7f4f7bfff700 1 -- 172.21.15.23:0/3456459583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f68005040 con 0x7f4f7c0f7510 2021-11-19T08:40:40.699 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.708+0000 7f4f7bfff700 1 --2- 172.21.15.23:0/3456459583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7c0f7510 0x7f4f7c1094c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4f68000c00 tx=0x7f4f6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:40.699 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.708+0000 7f4f797fa700 1 -- 172.21.15.23:0/3456459583 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f68019070 con 0x7f4f7c0f7510 2021-11-19T08:40:40.699 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.709+0000 7f4f797fa700 1 -- 172.21.15.23:0/3456459583 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f68007e70 con 0x7f4f7c0f7510 2021-11-19T08:40:40.700 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.709+0000 7f4f82b44700 1 -- 172.21.15.23:0/3456459583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7c10e5f0 con 0x7f4f7c0f7510 2021-11-19T08:40:40.700 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.709+0000 7f4f797fa700 1 -- 172.21.15.23:0/3456459583 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f68014c90 con 0x7f4f7c0f7510 2021-11-19T08:40:40.700 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.709+0000 7f4f82b44700 1 -- 172.21.15.23:0/3456459583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7c10ebb0 con 0x7f4f7c0f7510 2021-11-19T08:40:40.701 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.710+0000 7f4f82b44700 1 -- 172.21.15.23:0/3456459583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7c027a00 con 0x7f4f7c0f7510 2021-11-19T08:40:40.702 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.710+0000 7f4f797fa700 1 -- 172.21.15.23:0/3456459583 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4f68025070 con 0x7f4f7c0f7510 2021-11-19T08:40:40.702 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.710+0000 7f4f797fa700 1 --2- 172.21.15.23:0/3456459583 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4f6c05c2a0 0x7f4f6c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:40.702 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.710+0000 7f4f797fa700 1 -- 172.21.15.23:0/3456459583 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f4f68076e00 con 0x7f4f7c0f7510 2021-11-19T08:40:40.702 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.710+0000 7f4f808e0700 1 --2- 172.21.15.23:0/3456459583 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4f6c05c2a0 0x7f4f6c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:40.703 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.711+0000 7f4f808e0700 1 --2- 172.21.15.23:0/3456459583 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4f6c05c2a0 0x7f4f6c05e760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f4f70009fd0 tx=0x7f4f7000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:40.703 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.714+0000 7f4f797fa700 1 -- 172.21.15.23:0/3456459583 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f68049b40 con 0x7f4f7c0f7510 2021-11-19T08:40:40.860 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.875+0000 7f4f82b44700 1 -- 172.21.15.23:0/3456459583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f7c101060 con 0x7f4f7c0f7510 2021-11-19T08:40:40.861 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.877+0000 7f4f797fa700 1 -- 172.21.15.23:0/3456459583 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+14093 (secure 0 0 0) 0x7f4f6804d160 con 0x7f4f7c0f7510 2021-11-19T08:40:40.862 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:40.862 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":41,"fsid":"908b41ee-4913-11ec-8c2c-001a4aab830c","created":"2021-11-19T08:35:14.952900+0000","modified":"2021-11-19T08:40:23.207735+0000","last_up_change":"2021-11-19T08:39:20.513380+0000","last_in_change":"2021-11-19T08:39:05.162873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-19T08:38:42.679405+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-19T08:40:19.691661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-19T08:40:20.621494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"55d41e72-4389-4078-ab49-915578ae7162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6803","nonce":2382765305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6805","nonce":2382765305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6809","nonce":2382765305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6807","nonce":2382765305}]},"public_addr":"172.21.15.23:6803/2382765305","cluster_addr":"172.21.15.23:6805/2382765305","heartbeat_back_addr":"172.21.15.23:6809/2382765305","heartbeat_front_addr":"172.21.15.23:6807/2382765305","state":["exists","up"]},{"osd":1,"uuid":"5b7aea87-24b7-43e6-9ade-8fef0e9e6dd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6811","nonce":1051412153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6813","nonce":1051412153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6817","nonce":1051412153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6815","nonce":1051412153}]},"public_addr":"172.21.15.23:6811/1051412153","cluster_addr":"172.21.15.23:6813/1051412153","heartbeat_back_addr":"172.21.15.23:6817/1051412153","heartbeat_front_addr":"172.21.15.23:6815/1051412153","state":["exists","up"]},{"osd":2,"uuid":"e21cf6ec-4aa2-43cc-934a-bbd8d6e6694c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6819","nonce":954281054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6821","nonce":954281054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6825","nonce":954281054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6823","nonce":954281054}]},"public_addr":"172.21.15.23:6819/954281054","cluster_addr":"172.21.15.23:6821/954281054","heartbeat_back_addr":"172.21.15.23:6825/954281054","heartbeat_front_addr":"172.21.15.23:6823/954281054","state":["exists","up"]},{"osd":3,"uuid":"276cb410-5d23-4cdf-a090-e777ebe4c82f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6801","nonce":2145743581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6803","nonce":2145743581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6807","nonce":2145743581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6805","nonce":2145743581}]},"public_addr":"172.21.15.134:6801/2145743581","cluster_addr":"172.21.15.134:6803/2145743581","heartbeat_back_addr":"172.21.15.134:6807/2145743581","heartbeat_front_addr":"172.21.15.134:6805/2145743581","state":["exists","up"]},{"osd":4,"uuid":"015b6da1-35ed-43f4-84c9-337d2b0e79bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6809","nonce":1867765886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6811","nonce":1867765886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6815","nonce":1867765886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6813","nonce":1867765886}]},"public_addr":"172.21.15.134:6809/1867765886","cluster_addr":"172.21.15.134:6811/1867765886","heartbeat_back_addr":"172.21.15.134:6815/1867765886","heartbeat_front_addr":"172.21.15.134:6813/1867765886","state":["exists","up"]},{"osd":5,"uuid":"86ec41d3-38df-4ee0-a320-402055f7895b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6817","nonce":3784199400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6819","nonce":3784199400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6823","nonce":3784199400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6821","nonce":3784199400}]},"public_addr":"172.21.15.134:6817/3784199400","cluster_addr":"172.21.15.134:6819/3784199400","heartbeat_back_addr":"172.21.15.134:6823/3784199400","heartbeat_front_addr":"172.21.15.134:6821/3784199400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:37:52.252795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:12.408238+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:31.280017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:47.181003+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:39:03.121900+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:39:19.383582+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/192042719":"2021-11-20T08:36:18.730761+0000","172.21.15.23:0/4146945986":"2021-11-20T08:36:18.730761+0000","172.21.15.23:6801/158654977":"2021-11-20T08:36:18.730761+0000","172.21.15.23:6800/158654977":"2021-11-20T08:36:18.730761+0000","172.21.15.23:0/992169182":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/1001097146":"2021-11-20T08:36:02.243722+0000","172.21.15.23:6800/1352671878":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/3039325239":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6801/1352671878":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/4131282839":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6800/945698991":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6801/945698991":"2021-11-20T08:35:42.544921+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-19T08:40:40.863 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.878+0000 7f4f82b44700 1 -- 172.21.15.23:0/3456459583 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4f6c05c2a0 msgr2=0x7f4f6c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:40.864 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.878+0000 7f4f82b44700 1 --2- 172.21.15.23:0/3456459583 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4f6c05c2a0 0x7f4f6c05e760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f4f70009fd0 tx=0x7f4f7000b040).stop 2021-11-19T08:40:40.864 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.878+0000 7f4f82b44700 1 -- 172.21.15.23:0/3456459583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7c0f7510 msgr2=0x7f4f7c1094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:40.864 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.878+0000 7f4f82b44700 1 --2- 172.21.15.23:0/3456459583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7c0f7510 0x7f4f7c1094c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4f68000c00 tx=0x7f4f6800b040).stop 2021-11-19T08:40:40.864 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.878+0000 7f4f82b44700 1 -- 172.21.15.23:0/3456459583 shutdown_connections 2021-11-19T08:40:40.865 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.878+0000 7f4f82b44700 1 --2- 172.21.15.23:0/3456459583 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f7c0f6b70 0x7f4f7c108fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:40.865 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.878+0000 7f4f82b44700 1 --2- 172.21.15.23:0/3456459583 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4f6c05c2a0 0x7f4f6c05e760 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:40.865 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.878+0000 7f4f82b44700 1 --2- 172.21.15.23:0/3456459583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7c0f7510 0x7f4f7c1094c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:40.865 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.878+0000 7f4f82b44700 1 -- 172.21.15.23:0/3456459583 >> 172.21.15.23:0/3456459583 conn(0x7f4f7c0f0440 msgr2=0x7f4f7c0fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:40.866 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.878+0000 7f4f82b44700 1 -- 172.21.15.23:0/3456459583 shutdown_connections 2021-11-19T08:40:40.866 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:40.879+0000 7f4f82b44700 1 -- 172.21.15.23:0/3456459583 wait complete. 2021-11-19T08:40:40.876 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-19T08:38:42.679405+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-19T08:40:19.691661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-19T08:40:20.621494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-19T08:40:40.877 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-19T08:40:41.001 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.015+0000 7fe2717bb700 1 -- 172.21.15.23:0/3538633923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe26c0f5580 msgr2=0x7fe26c0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:41.001 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.015+0000 7fe2717bb700 1 --2- 172.21.15.23:0/3538633923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe26c0f5580 0x7fe26c0f59a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fe25c004660 tx=0x7fe25c009b30).stop 2021-11-19T08:40:41.001 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.015+0000 7fe2717bb700 1 -- 172.21.15.23:0/3538633923 shutdown_connections 2021-11-19T08:40:41.002 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.015+0000 7fe2717bb700 1 --2- 172.21.15.23:0/3538633923 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe26c0f6820 0x7fe26c0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.002 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.015+0000 7fe2717bb700 1 --2- 172.21.15.23:0/3538633923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe26c0f5580 0x7fe26c0f59a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.002 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.015+0000 7fe2717bb700 1 -- 172.21.15.23:0/3538633923 >> 172.21.15.23:0/3538633923 conn(0x7fe26c0f0ce0 msgr2=0x7fe26c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:41.002 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.015+0000 7fe2717bb700 1 -- 172.21.15.23:0/3538633923 shutdown_connections 2021-11-19T08:40:41.003 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.016+0000 7fe2717bb700 1 -- 172.21.15.23:0/3538633923 wait complete. 2021-11-19T08:40:41.005 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.017+0000 7fe2717bb700 1 Processor -- start 2021-11-19T08:40:41.005 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.017+0000 7fe2717bb700 1 -- start start 2021-11-19T08:40:41.006 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.017+0000 7fe2717bb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe26c0f5580 0x7fe26c060420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:41.006 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.017+0000 7fe2717bb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe26c0f6820 0x7fe26c10bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:41.006 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.017+0000 7fe2717bb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c10c1a0 con 0x7fe26c0f6820 2021-11-19T08:40:41.006 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.017+0000 7fe2717bb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c10c2e0 con 0x7fe26c0f5580 2021-11-19T08:40:41.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.017+0000 7fe26a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe26c0f6820 0x7fe26c10bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:41.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.017+0000 7fe26a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe26c0f6820 0x7fe26c10bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47060/0 (socket says 172.21.15.23:47060) 2021-11-19T08:40:41.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.017+0000 7fe26a7fc700 1 -- 172.21.15.23:0/2543929679 learned_addr learned my addr 172.21.15.23:0/2543929679 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:41.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.017+0000 7fe26affd700 1 --2- 172.21.15.23:0/2543929679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe26c0f5580 0x7fe26c060420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:41.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.018+0000 7fe26a7fc700 1 -- 172.21.15.23:0/2543929679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe26c0f5580 msgr2=0x7fe26c060420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:41.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.018+0000 7fe26a7fc700 1 --2- 172.21.15.23:0/2543929679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe26c0f5580 0x7fe26c060420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.018+0000 7fe26a7fc700 1 -- 172.21.15.23:0/2543929679 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c005040 con 0x7fe26c0f6820 2021-11-19T08:40:41.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.018+0000 7fe26a7fc700 1 --2- 172.21.15.23:0/2543929679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe26c0f6820 0x7fe26c10bcb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fe2600026d0 tx=0x7fe260009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:41.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.018+0000 7fe2537fe700 1 -- 172.21.15.23:0/2543929679 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2600079c0 con 0x7fe26c0f6820 2021-11-19T08:40:41.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.018+0000 7fe2717bb700 1 -- 172.21.15.23:0/2543929679 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c10c540 con 0x7fe26c0f6820 2021-11-19T08:40:41.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.018+0000 7fe2717bb700 1 -- 172.21.15.23:0/2543929679 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c10cad0 con 0x7fe26c0f6820 2021-11-19T08:40:41.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.018+0000 7fe2537fe700 1 -- 172.21.15.23:0/2543929679 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe26000f040 con 0x7fe26c0f6820 2021-11-19T08:40:41.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.018+0000 7fe2537fe700 1 -- 172.21.15.23:0/2543929679 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe26001d7d0 con 0x7fe26c0f6820 2021-11-19T08:40:41.010 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.020+0000 7fe2717bb700 1 -- 172.21.15.23:0/2543929679 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe24c004fa0 con 0x7fe26c0f6820 2021-11-19T08:40:41.010 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.020+0000 7fe2537fe700 1 -- 172.21.15.23:0/2543929679 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe26001d930 con 0x7fe26c0f6820 2021-11-19T08:40:41.010 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.020+0000 7fe2537fe700 1 --2- 172.21.15.23:0/2543929679 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe25405c2a0 0x7fe25405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:41.010 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.020+0000 7fe2537fe700 1 -- 172.21.15.23:0/2543929679 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fe260075dd0 con 0x7fe26c0f6820 2021-11-19T08:40:41.010 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.021+0000 7fe26affd700 1 --2- 172.21.15.23:0/2543929679 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe25405c2a0 0x7fe25405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:41.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.021+0000 7fe26affd700 1 --2- 172.21.15.23:0/2543929679 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe25405c2a0 0x7fe25405e760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe25c009f50 tx=0x7fe25c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:41.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.026+0000 7fe2537fe700 1 -- 172.21.15.23:0/2543929679 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe26004c4c0 con 0x7fe26c0f6820 2021-11-19T08:40:41.173 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.188+0000 7fe2717bb700 1 -- 172.21.15.23:0/2543929679 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe24c005bf0 con 0x7fe26c0f6820 2021-11-19T08:40:41.174 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.189+0000 7fe2537fe700 1 -- 172.21.15.23:0/2543929679 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v41) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe26004c0b0 con 0x7fe26c0f6820 2021-11-19T08:40:41.174 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2021-11-19T08:40:41.175 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.190+0000 7fe2717bb700 1 -- 172.21.15.23:0/2543929679 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe25405c2a0 msgr2=0x7fe25405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:41.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.190+0000 7fe2717bb700 1 --2- 172.21.15.23:0/2543929679 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe25405c2a0 0x7fe25405e760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe25c009f50 tx=0x7fe25c009b30).stop 2021-11-19T08:40:41.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.190+0000 7fe2717bb700 1 -- 172.21.15.23:0/2543929679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe26c0f6820 msgr2=0x7fe26c10bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:41.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.190+0000 7fe2717bb700 1 --2- 172.21.15.23:0/2543929679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe26c0f6820 0x7fe26c10bcb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fe2600026d0 tx=0x7fe260009de0).stop 2021-11-19T08:40:41.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.190+0000 7fe2717bb700 1 -- 172.21.15.23:0/2543929679 shutdown_connections 2021-11-19T08:40:41.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.190+0000 7fe2717bb700 1 --2- 172.21.15.23:0/2543929679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe26c0f5580 0x7fe26c060420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.190+0000 7fe2717bb700 1 --2- 172.21.15.23:0/2543929679 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe25405c2a0 0x7fe25405e760 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.190+0000 7fe2717bb700 1 --2- 172.21.15.23:0/2543929679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe26c0f6820 0x7fe26c10bcb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.190+0000 7fe2717bb700 1 -- 172.21.15.23:0/2543929679 >> 172.21.15.23:0/2543929679 conn(0x7fe26c0f0ce0 msgr2=0x7fe26c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:41.178 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.191+0000 7fe2717bb700 1 -- 172.21.15.23:0/2543929679 shutdown_connections 2021-11-19T08:40:41.178 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.191+0000 7fe2717bb700 1 -- 172.21.15.23:0/2543929679 wait complete. 2021-11-19T08:40:41.192 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-19T08:40:41.321 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.334+0000 7fcc10a5a700 1 -- 172.21.15.23:0/4195510012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc0c0f6ba0 msgr2=0x7fcc0c0f6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:41.321 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.334+0000 7fcc10a5a700 1 --2- 172.21.15.23:0/4195510012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc0c0f6ba0 0x7fcc0c0f6fc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fcbfc004660 tx=0x7fcbfc009b30).stop 2021-11-19T08:40:41.322 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.334+0000 7fcc10a5a700 1 -- 172.21.15.23:0/4195510012 shutdown_connections 2021-11-19T08:40:41.322 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.334+0000 7fcc10a5a700 1 --2- 172.21.15.23:0/4195510012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0c0f7540 0x7fcc0c0f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.322 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.334+0000 7fcc10a5a700 1 --2- 172.21.15.23:0/4195510012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc0c0f6ba0 0x7fcc0c0f6fc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.322 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.334+0000 7fcc10a5a700 1 -- 172.21.15.23:0/4195510012 >> 172.21.15.23:0/4195510012 conn(0x7fcc0c0f04a0 msgr2=0x7fcc0c0f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:41.323 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.334+0000 7fcc10a5a700 1 -- 172.21.15.23:0/4195510012 shutdown_connections 2021-11-19T08:40:41.323 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.334+0000 7fcc10a5a700 1 -- 172.21.15.23:0/4195510012 wait complete. 2021-11-19T08:40:41.323 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.334+0000 7fcc10a5a700 1 Processor -- start 2021-11-19T08:40:41.323 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.334+0000 7fcc10a5a700 1 -- start start 2021-11-19T08:40:41.324 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc10a5a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc0c0f6ba0 0x7fcc0c0f5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:41.324 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc10a5a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0c0f7540 0x7fcc0c10e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:41.324 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc10a5a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c107ad0 con 0x7fcc0c0f6ba0 2021-11-19T08:40:41.324 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc10a5a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c10e5e0 con 0x7fcc0c0f7540 2021-11-19T08:40:41.325 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc0a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc0c0f6ba0 0x7fcc0c0f5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:41.325 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc0a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc0c0f6ba0 0x7fcc0c0f5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47068/0 (socket says 172.21.15.23:47068) 2021-11-19T08:40:41.325 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc0a59c700 1 -- 172.21.15.23:0/167695236 learned_addr learned my addr 172.21.15.23:0/167695236 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:41.325 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc09d9b700 1 --2- 172.21.15.23:0/167695236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0c0f7540 0x7fcc0c10e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:41.325 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc0a59c700 1 -- 172.21.15.23:0/167695236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0c0f7540 msgr2=0x7fcc0c10e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:41.326 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc0a59c700 1 --2- 172.21.15.23:0/167695236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0c0f7540 0x7fcc0c10e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.326 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc0a59c700 1 -- 172.21.15.23:0/167695236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbfc005040 con 0x7fcc0c0f6ba0 2021-11-19T08:40:41.326 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc09d9b700 1 --2- 172.21.15.23:0/167695236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0c0f7540 0x7fcc0c10e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:40:41.326 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc0a59c700 1 --2- 172.21.15.23:0/167695236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc0c0f6ba0 0x7fcc0c0f5330 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fcbfc004e30 tx=0x7fcbfc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:41.327 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcbfaffd700 1 -- 172.21.15.23:0/167695236 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbfc018070 con 0x7fcc0c0f6ba0 2021-11-19T08:40:41.327 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcbfaffd700 1 -- 172.21.15.23:0/167695236 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbfc013410 con 0x7fcc0c0f6ba0 2021-11-19T08:40:41.327 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc10a5a700 1 -- 172.21.15.23:0/167695236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0c10e720 con 0x7fcc0c0f6ba0 2021-11-19T08:40:41.327 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcbfaffd700 1 -- 172.21.15.23:0/167695236 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbfc01c3c0 con 0x7fcc0c0f6ba0 2021-11-19T08:40:41.328 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.335+0000 7fcc10a5a700 1 -- 172.21.15.23:0/167695236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0c10ec50 con 0x7fcc0c0f6ba0 2021-11-19T08:40:41.328 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.336+0000 7fcc10a5a700 1 -- 172.21.15.23:0/167695236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0c027a00 con 0x7fcc0c0f6ba0 2021-11-19T08:40:41.329 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.337+0000 7fcbfaffd700 1 -- 172.21.15.23:0/167695236 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcbfc01d420 con 0x7fcc0c0f6ba0 2021-11-19T08:40:41.329 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.337+0000 7fcbfaffd700 1 --2- 172.21.15.23:0/167695236 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcbf405c2a0 0x7fcbf405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:41.329 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.337+0000 7fcbfaffd700 1 -- 172.21.15.23:0/167695236 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fcbfc077b60 con 0x7fcc0c0f6ba0 2021-11-19T08:40:41.330 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.337+0000 7fcc09d9b700 1 --2- 172.21.15.23:0/167695236 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcbf405c2a0 0x7fcbf405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:41.330 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.338+0000 7fcc09d9b700 1 --2- 172.21.15.23:0/167695236 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcbf405c2a0 0x7fcbf405e760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fcc00009fd0 tx=0x7fcc0000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:41.330 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.341+0000 7fcbfaffd700 1 -- 172.21.15.23:0/167695236 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcbfc04a820 con 0x7fcc0c0f6ba0 2021-11-19T08:40:41.492 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.507+0000 7fcc10a5a700 1 -- 172.21.15.23:0/167695236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fcc0c0fdd90 con 0x7fcc0c0f6ba0 2021-11-19T08:40:41.493 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.508+0000 7fcbfaffd700 1 -- 172.21.15.23:0/167695236 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7fcbfc04de40 con 0x7fcc0c0f6ba0 2021-11-19T08:40:41.493 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2021-11-19T08:40:41.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.510+0000 7fcc10a5a700 1 -- 172.21.15.23:0/167695236 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcbf405c2a0 msgr2=0x7fcbf405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:41.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.510+0000 7fcc10a5a700 1 --2- 172.21.15.23:0/167695236 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcbf405c2a0 0x7fcbf405e760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fcc00009fd0 tx=0x7fcc0000b040).stop 2021-11-19T08:40:41.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.510+0000 7fcc10a5a700 1 -- 172.21.15.23:0/167695236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc0c0f6ba0 msgr2=0x7fcc0c0f5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:41.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.510+0000 7fcc10a5a700 1 --2- 172.21.15.23:0/167695236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc0c0f6ba0 0x7fcc0c0f5330 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fcbfc004e30 tx=0x7fcbfc009b30).stop 2021-11-19T08:40:41.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.510+0000 7fcc10a5a700 1 -- 172.21.15.23:0/167695236 shutdown_connections 2021-11-19T08:40:41.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.510+0000 7fcc10a5a700 1 --2- 172.21.15.23:0/167695236 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcbf405c2a0 0x7fcbf405e760 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.510+0000 7fcc10a5a700 1 --2- 172.21.15.23:0/167695236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc0c0f6ba0 0x7fcc0c0f5330 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.497 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.510+0000 7fcc10a5a700 1 --2- 172.21.15.23:0/167695236 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0c0f7540 0x7fcc0c10e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.497 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.510+0000 7fcc10a5a700 1 -- 172.21.15.23:0/167695236 >> 172.21.15.23:0/167695236 conn(0x7fcc0c0f04a0 msgr2=0x7fcc0c0fa3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:41.497 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.510+0000 7fcc10a5a700 1 -- 172.21.15.23:0/167695236 shutdown_connections 2021-11-19T08:40:41.497 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.510+0000 7fcc10a5a700 1 -- 172.21.15.23:0/167695236 wait complete. 2021-11-19T08:40:41.507 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:41 smithi023 conmon[33490]: audit 2021-11-19T08:40:40.877401+0000 mon.smithi023 (mon.0) 536 : audit [DBG] from='client.? 172.21.15.23:0/3456459583' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-19T08:40:41.507 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:41 smithi023 conmon[33490]: audit 2021-11-19T08:40:41.189782+0000 mon.smithi023 (mon.0) 537 : audit [DBG] from='client.? 172.21.15.23:0/2543929679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-19T08:40:41.509 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-19T08:40:41.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:41 smithi134 conmon[36363]: audit 2021-11-19T08:40:40.877401+0000 mon.smithi023 (mon.0) 536 : audit [DBG] from='client.? 172.21.15.23:0/3456459583' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-19T08:40:41.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:41 smithi134 conmon[36363]: audit 2021-11-19T08:40:41.189782+0000 mon.smithi023 (mon.0) 537 : audit [DBG] from='client.? 172.21.15.23:0/2543929679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-19T08:40:41.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.645+0000 7fb411f0e700 1 -- 172.21.15.23:0/1114532771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb40c0f7390 msgr2=0x7fb40c0f57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:41.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.645+0000 7fb411f0e700 1 --2- 172.21.15.23:0/1114532771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb40c0f7390 0x7fb40c0f57b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fb3f8004660 tx=0x7fb3f8009b30).stop 2021-11-19T08:40:41.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.645+0000 7fb411f0e700 1 -- 172.21.15.23:0/1114532771 shutdown_connections 2021-11-19T08:40:41.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.645+0000 7fb411f0e700 1 --2- 172.21.15.23:0/1114532771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb40c0f7390 0x7fb40c0f57b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.645+0000 7fb411f0e700 1 --2- 172.21.15.23:0/1114532771 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb40c0f69f0 0x7fb40c0f6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.645+0000 7fb411f0e700 1 -- 172.21.15.23:0/1114532771 >> 172.21.15.23:0/1114532771 conn(0x7fb40c0f02c0 msgr2=0x7fb40c0f26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:41.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.645+0000 7fb411f0e700 1 -- 172.21.15.23:0/1114532771 shutdown_connections 2021-11-19T08:40:41.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.645+0000 7fb411f0e700 1 -- 172.21.15.23:0/1114532771 wait complete. 2021-11-19T08:40:41.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.646+0000 7fb411f0e700 1 Processor -- start 2021-11-19T08:40:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.646+0000 7fb411f0e700 1 -- start start 2021-11-19T08:40:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.646+0000 7fb411f0e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb40c0f69f0 0x7fb40c109a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.646+0000 7fb411f0e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb40c0f7390 0x7fb40c10bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.646+0000 7fb411f0e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c0f5480 con 0x7fb40c0f69f0 2021-11-19T08:40:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.646+0000 7fb411f0e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c0f55c0 con 0x7fb40c0f7390 2021-11-19T08:40:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.646+0000 7fb410f0c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb40c0f69f0 0x7fb40c109a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.646+0000 7fb410f0c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb40c0f69f0 0x7fb40c109a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47078/0 (socket says 172.21.15.23:47078) 2021-11-19T08:40:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.646+0000 7fb410f0c700 1 -- 172.21.15.23:0/2088051968 learned_addr learned my addr 172.21.15.23:0/2088051968 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.647+0000 7fb40bfff700 1 --2- 172.21.15.23:0/2088051968 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb40c0f7390 0x7fb40c10bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:41.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.647+0000 7fb410f0c700 1 -- 172.21.15.23:0/2088051968 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb40c0f7390 msgr2=0x7fb40c10bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:41.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.647+0000 7fb410f0c700 1 --2- 172.21.15.23:0/2088051968 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb40c0f7390 0x7fb40c10bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.647+0000 7fb410f0c700 1 -- 172.21.15.23:0/2088051968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f8005040 con 0x7fb40c0f69f0 2021-11-19T08:40:41.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.647+0000 7fb40bfff700 1 --2- 172.21.15.23:0/2088051968 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb40c0f7390 0x7fb40c10bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:40:41.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.647+0000 7fb410f0c700 1 --2- 172.21.15.23:0/2088051968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb40c0f69f0 0x7fb40c109a50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fb400002700 tx=0x7fb400009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:41.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.647+0000 7fb4097fa700 1 -- 172.21.15.23:0/2088051968 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb400009320 con 0x7fb40c0f69f0 2021-11-19T08:40:41.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.647+0000 7fb4097fa700 1 -- 172.21.15.23:0/2088051968 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb40000f040 con 0x7fb40c0f69f0 2021-11-19T08:40:41.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.647+0000 7fb4097fa700 1 -- 172.21.15.23:0/2088051968 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb400014900 con 0x7fb40c0f69f0 2021-11-19T08:40:41.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.647+0000 7fb411f0e700 1 -- 172.21.15.23:0/2088051968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb40c10c560 con 0x7fb40c0f69f0 2021-11-19T08:40:41.639 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.647+0000 7fb411f0e700 1 -- 172.21.15.23:0/2088051968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40c10caf0 con 0x7fb40c0f69f0 2021-11-19T08:40:41.639 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.648+0000 7fb411f0e700 1 -- 172.21.15.23:0/2088051968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40c0f3f50 con 0x7fb40c0f69f0 2021-11-19T08:40:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.648+0000 7fb4097fa700 1 -- 172.21.15.23:0/2088051968 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb400009480 con 0x7fb40c0f69f0 2021-11-19T08:40:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.648+0000 7fb4097fa700 1 --2- 172.21.15.23:0/2088051968 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb3fc05c2b0 0x7fb3fc05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.649+0000 7fb4097fa700 1 -- 172.21.15.23:0/2088051968 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fb400076290 con 0x7fb40c0f69f0 2021-11-19T08:40:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.649+0000 7fb40bfff700 1 --2- 172.21.15.23:0/2088051968 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb3fc05c2b0 0x7fb3fc05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.649+0000 7fb40bfff700 1 --2- 172.21.15.23:0/2088051968 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb3fc05c2b0 0x7fb3fc05e770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fb3f8009d60 tx=0x7fb3f8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.652+0000 7fb4097fa700 1 -- 172.21.15.23:0/2088051968 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb400048e90 con 0x7fb40c0f69f0 2021-11-19T08:40:41.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.818+0000 7fb411f0e700 1 -- 172.21.15.23:0/2088051968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fb40c045ad0 con 0x7fb40c0f69f0 2021-11-19T08:40:41.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.818+0000 7fb4097fa700 1 -- 172.21.15.23:0/2088051968 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb40004c4b0 con 0x7fb40c0f69f0 2021-11-19T08:40:41.804 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2021-11-19T08:40:41.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.820+0000 7fb411f0e700 1 -- 172.21.15.23:0/2088051968 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb3fc05c2b0 msgr2=0x7fb3fc05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:41.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.820+0000 7fb411f0e700 1 --2- 172.21.15.23:0/2088051968 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb3fc05c2b0 0x7fb3fc05e770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fb3f8009d60 tx=0x7fb3f8009b30).stop 2021-11-19T08:40:41.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.820+0000 7fb411f0e700 1 -- 172.21.15.23:0/2088051968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb40c0f69f0 msgr2=0x7fb40c109a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:41.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.820+0000 7fb411f0e700 1 --2- 172.21.15.23:0/2088051968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb40c0f69f0 0x7fb40c109a50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fb400002700 tx=0x7fb400009d80).stop 2021-11-19T08:40:41.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.820+0000 7fb411f0e700 1 -- 172.21.15.23:0/2088051968 shutdown_connections 2021-11-19T08:40:41.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.820+0000 7fb411f0e700 1 --2- 172.21.15.23:0/2088051968 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb3fc05c2b0 0x7fb3fc05e770 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.820+0000 7fb411f0e700 1 --2- 172.21.15.23:0/2088051968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb40c0f69f0 0x7fb40c109a50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.820+0000 7fb411f0e700 1 --2- 172.21.15.23:0/2088051968 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb40c0f7390 0x7fb40c10bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.820+0000 7fb411f0e700 1 -- 172.21.15.23:0/2088051968 >> 172.21.15.23:0/2088051968 conn(0x7fb40c0f02c0 msgr2=0x7fb40c0fa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:41.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.820+0000 7fb411f0e700 1 -- 172.21.15.23:0/2088051968 shutdown_connections 2021-11-19T08:40:41.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.820+0000 7fb411f0e700 1 -- 172.21.15.23:0/2088051968 wait complete. 2021-11-19T08:40:41.817 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-11-19T08:40:41.818 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:40:41.943 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.956+0000 7fecddfe6700 1 -- 172.21.15.23:0/1112459058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd80f55a0 msgr2=0x7fecd80f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:41.943 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.956+0000 7fecddfe6700 1 --2- 172.21.15.23:0/1112459058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd80f55a0 0x7fecd80f59c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fecc8004660 tx=0x7fecc8009b30).stop 2021-11-19T08:40:41.945 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.956+0000 7fecddfe6700 1 -- 172.21.15.23:0/1112459058 shutdown_connections 2021-11-19T08:40:41.945 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.956+0000 7fecddfe6700 1 --2- 172.21.15.23:0/1112459058 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fecd80f6840 0x7fecd80f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.945 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.956+0000 7fecddfe6700 1 --2- 172.21.15.23:0/1112459058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd80f55a0 0x7fecd80f59c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.946 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.956+0000 7fecddfe6700 1 -- 172.21.15.23:0/1112459058 >> 172.21.15.23:0/1112459058 conn(0x7fecd80f0ce0 msgr2=0x7fecd80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:41.946 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.956+0000 7fecddfe6700 1 -- 172.21.15.23:0/1112459058 shutdown_connections 2021-11-19T08:40:41.946 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.956+0000 7fecddfe6700 1 -- 172.21.15.23:0/1112459058 wait complete. 2021-11-19T08:40:41.946 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.957+0000 7fecddfe6700 1 Processor -- start 2021-11-19T08:40:41.947 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.957+0000 7fecddfe6700 1 -- start start 2021-11-19T08:40:41.947 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.957+0000 7fecddfe6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd80f55a0 0x7fecd805ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:41.947 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.957+0000 7fecddfe6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fecd80f6840 0x7fecd810bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:41.947 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.957+0000 7fecddfe6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd810c1d0 con 0x7fecd80f55a0 2021-11-19T08:40:41.948 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.957+0000 7fecddfe6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd810c310 con 0x7fecd80f6840 2021-11-19T08:40:41.948 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.957+0000 7fecd77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd80f55a0 0x7fecd805ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:41.948 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.957+0000 7fecd77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd80f55a0 0x7fecd805ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47088/0 (socket says 172.21.15.23:47088) 2021-11-19T08:40:41.948 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.957+0000 7fecd77fe700 1 -- 172.21.15.23:0/738411247 learned_addr learned my addr 172.21.15.23:0/738411247 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:41.948 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.957+0000 7fecd6ffd700 1 --2- 172.21.15.23:0/738411247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fecd80f6840 0x7fecd810bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:41.949 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.957+0000 7fecd77fe700 1 -- 172.21.15.23:0/738411247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fecd80f6840 msgr2=0x7fecd810bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:41.949 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.957+0000 7fecd77fe700 1 --2- 172.21.15.23:0/738411247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fecd80f6840 0x7fecd810bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:41.949 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.957+0000 7fecd77fe700 1 -- 172.21.15.23:0/738411247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc8005040 con 0x7fecd80f55a0 2021-11-19T08:40:41.949 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.957+0000 7fecd77fe700 1 --2- 172.21.15.23:0/738411247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd80f55a0 0x7fecd805ff00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fecc8004e30 tx=0x7fecc8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:41.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.958+0000 7fecbffff700 1 -- 172.21.15.23:0/738411247 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecc8018070 con 0x7fecd80f55a0 2021-11-19T08:40:41.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.958+0000 7fecddfe6700 1 -- 172.21.15.23:0/738411247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd810c510 con 0x7fecd80f55a0 2021-11-19T08:40:41.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.958+0000 7fecbffff700 1 -- 172.21.15.23:0/738411247 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecc8013410 con 0x7fecd80f55a0 2021-11-19T08:40:41.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.958+0000 7fecbffff700 1 -- 172.21.15.23:0/738411247 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecc801c3c0 con 0x7fecd80f55a0 2021-11-19T08:40:41.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.958+0000 7fecddfe6700 1 -- 172.21.15.23:0/738411247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd810ca40 con 0x7fecd80f55a0 2021-11-19T08:40:41.951 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.958+0000 7fecddfe6700 1 -- 172.21.15.23:0/738411247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd8027a00 con 0x7fecd80f55a0 2021-11-19T08:40:41.952 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.959+0000 7fecbffff700 1 -- 172.21.15.23:0/738411247 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fecc801d420 con 0x7fecd80f55a0 2021-11-19T08:40:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.959+0000 7fecbffff700 1 --2- 172.21.15.23:0/738411247 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fecc005c2a0 0x7fecc005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.959+0000 7fecbffff700 1 -- 172.21.15.23:0/738411247 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fecc8077ac0 con 0x7fecd80f55a0 2021-11-19T08:40:41.954 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.959+0000 7fecd6ffd700 1 --2- 172.21.15.23:0/738411247 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fecc005c2a0 0x7fecc005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:41.954 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.960+0000 7fecd6ffd700 1 --2- 172.21.15.23:0/738411247 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fecc005c2a0 0x7fecc005e760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7feccc0096c0 tx=0x7feccc009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:41.954 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:41.963+0000 7fecbffff700 1 -- 172.21.15.23:0/738411247 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fecc804a780 con 0x7fecd80f55a0 2021-11-19T08:40:42.149 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.164+0000 7fecddfe6700 1 -- 172.21.15.23:0/738411247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecd80fdd30 con 0x7fecd80f55a0 2021-11-19T08:40:42.149 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.165+0000 7fecbffff700 1 -- 172.21.15.23:0/738411247 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2962 (secure 0 0 0) 0x7fecc804dda0 con 0x7fecd80f55a0 2021-11-19T08:40:42.150 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:42.150 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":14,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:40:37.178115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14434,"mds_1":24283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14434":{"gid":14434,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.23:6827/530484646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":530484646},{"type":"v1","addr":"172.21.15.23:6827","nonce":530484646}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14446":{"gid":14446,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.23:6829/2176063367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2176063367},{"type":"v1","addr":"172.21.15.23:6829","nonce":2176063367}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24271":{"gid":24271,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.134:6825/1874302373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1874302373},{"type":"v1","addr":"172.21.15.134:6825","nonce":1874302373}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24283":{"gid":24283,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":8,"state":"up:active","state_seq":2,"addr":"172.21.15.134:6827/3905671987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3905671987},{"type":"v1","addr":"172.21.15.134:6827","nonce":3905671987}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:40:42.151 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.166+0000 7fecddfe6700 1 -- 172.21.15.23:0/738411247 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fecc005c2a0 msgr2=0x7fecc005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.152 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.166+0000 7fecddfe6700 1 --2- 172.21.15.23:0/738411247 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fecc005c2a0 0x7fecc005e760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7feccc0096c0 tx=0x7feccc009be0).stop 2021-11-19T08:40:42.152 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.166+0000 7fecddfe6700 1 -- 172.21.15.23:0/738411247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd80f55a0 msgr2=0x7fecd805ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.152 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.166+0000 7fecddfe6700 1 --2- 172.21.15.23:0/738411247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd80f55a0 0x7fecd805ff00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fecc8004e30 tx=0x7fecc8009b30).stop 2021-11-19T08:40:42.153 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.167+0000 7fecddfe6700 1 -- 172.21.15.23:0/738411247 shutdown_connections 2021-11-19T08:40:42.153 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.167+0000 7fecddfe6700 1 --2- 172.21.15.23:0/738411247 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fecc005c2a0 0x7fecc005e760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.153 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.167+0000 7fecddfe6700 1 --2- 172.21.15.23:0/738411247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecd80f55a0 0x7fecd805ff00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.153 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.167+0000 7fecddfe6700 1 --2- 172.21.15.23:0/738411247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fecd80f6840 0x7fecd810bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.153 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.167+0000 7fecddfe6700 1 -- 172.21.15.23:0/738411247 >> 172.21.15.23:0/738411247 conn(0x7fecd80f0ce0 msgr2=0x7fecd80fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:42.154 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.167+0000 7fecddfe6700 1 -- 172.21.15.23:0/738411247 shutdown_connections 2021-11-19T08:40:42.154 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.167+0000 7fecddfe6700 1 -- 172.21.15.23:0/738411247 wait complete. 2021-11-19T08:40:42.154 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 14 2021-11-19T08:40:42.164 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:40:37.178115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14434, 'mds_1': 24283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14434': {'gid': 14434, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.23:6827/530484646', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 530484646}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 530484646}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14446': {'gid': 14446, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 3, 'addr': '172.21.15.23:6829/2176063367', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 2176063367}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 2176063367}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24271': {'gid': 24271, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 3, 'addr': '172.21.15.134:6825/1874302373', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 1874302373}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 1874302373}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24283': {'gid': 24283, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 8, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.134:6827/3905671987', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3905671987}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3905671987}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:40:42.165 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-11-19T08:40:42.165 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.cephfs.smithi023.fuvzkr status 2021-11-19T08:40:42.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.304+0000 7fc6d54b7700 1 -- 172.21.15.23:0/2527897513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d00f6690 msgr2=0x7fc6d00f6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.304+0000 7fc6d54b7700 1 --2- 172.21.15.23:0/2527897513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d00f6690 0x7fc6d00f6af0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fc6c4004660 tx=0x7fc6c4009b30).stop 2021-11-19T08:40:42.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.304+0000 7fc6d54b7700 1 -- 172.21.15.23:0/2527897513 shutdown_connections 2021-11-19T08:40:42.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.304+0000 7fc6d54b7700 1 --2- 172.21.15.23:0/2527897513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d00f6690 0x7fc6d00f6af0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.304+0000 7fc6d54b7700 1 --2- 172.21.15.23:0/2527897513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d00f53f0 0x7fc6d00f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.304+0000 7fc6d54b7700 1 -- 172.21.15.23:0/2527897513 >> 172.21.15.23:0/2527897513 conn(0x7fc6d00f0b30 msgr2=0x7fc6d00f2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:42.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.304+0000 7fc6d54b7700 1 -- 172.21.15.23:0/2527897513 shutdown_connections 2021-11-19T08:40:42.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.304+0000 7fc6d54b7700 1 -- 172.21.15.23:0/2527897513 wait complete. 2021-11-19T08:40:42.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.305+0000 7fc6d54b7700 1 Processor -- start 2021-11-19T08:40:42.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.305+0000 7fc6d54b7700 1 -- start start 2021-11-19T08:40:42.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.305+0000 7fc6d54b7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d00f53f0 0x7fc6d0106d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.305+0000 7fc6d54b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d00f6690 0x7fc6d0107280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.305+0000 7fc6d54b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d0107880 con 0x7fc6d00f6690 2021-11-19T08:40:42.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.305+0000 7fc6d54b7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d0108090 con 0x7fc6d00f53f0 2021-11-19T08:40:42.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.305+0000 7fc6cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d00f6690 0x7fc6d0107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.305+0000 7fc6cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d00f6690 0x7fc6d0107280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47100/0 (socket says 172.21.15.23:47100) 2021-11-19T08:40:42.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.305+0000 7fc6cf7fe700 1 -- 172.21.15.23:0/2146037625 learned_addr learned my addr 172.21.15.23:0/2146037625 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:42.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.306+0000 7fc6cffff700 1 --2- 172.21.15.23:0/2146037625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d00f53f0 0x7fc6d0106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.306+0000 7fc6cf7fe700 1 -- 172.21.15.23:0/2146037625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d00f53f0 msgr2=0x7fc6d0106d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.306+0000 7fc6cf7fe700 1 --2- 172.21.15.23:0/2146037625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d00f53f0 0x7fc6d0106d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.306+0000 7fc6cf7fe700 1 -- 172.21.15.23:0/2146037625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c4005040 con 0x7fc6d00f6690 2021-11-19T08:40:42.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.306+0000 7fc6cffff700 1 --2- 172.21.15.23:0/2146037625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d00f53f0 0x7fc6d0106d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:40:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.306+0000 7fc6cf7fe700 1 --2- 172.21.15.23:0/2146037625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d00f6690 0x7fc6d0107280 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fc6c4000c00 tx=0x7fc6c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.306+0000 7fc6ccff9700 1 -- 172.21.15.23:0/2146037625 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c4019070 con 0x7fc6d00f6690 2021-11-19T08:40:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.306+0000 7fc6ccff9700 1 -- 172.21.15.23:0/2146037625 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6c4007e70 con 0x7fc6d00f6690 2021-11-19T08:40:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.306+0000 7fc6d54b7700 1 -- 172.21.15.23:0/2146037625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d01082b0 con 0x7fc6d00f6690 2021-11-19T08:40:42.298 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.306+0000 7fc6ccff9700 1 -- 172.21.15.23:0/2146037625 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c4014d00 con 0x7fc6d00f6690 2021-11-19T08:40:42.298 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.306+0000 7fc6d54b7700 1 -- 172.21.15.23:0/2146037625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d0108810 con 0x7fc6d00f6690 2021-11-19T08:40:42.305 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.307+0000 7fc6ccff9700 1 -- 172.21.15.23:0/2146037625 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc6c4025070 con 0x7fc6d00f6690 2021-11-19T08:40:42.305 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.308+0000 7fc6ccff9700 1 --2- 172.21.15.23:0/2146037625 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc6b805c2b0 0x7fc6b805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.305 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.308+0000 7fc6ccff9700 1 -- 172.21.15.23:0/2146037625 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fc6c4076f30 con 0x7fc6d00f6690 2021-11-19T08:40:42.305 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.308+0000 7fc6cffff700 1 --2- 172.21.15.23:0/2146037625 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc6b805c2b0 0x7fc6b805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.306 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.308+0000 7fc6cffff700 1 --2- 172.21.15.23:0/2146037625 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc6b805c2b0 0x7fc6b805e770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fc6c0009da0 tx=0x7fc6c000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:42.306 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.309+0000 7fc6e82c4b80 1 Processor -- start 2021-11-19T08:40:42.306 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.309+0000 7fc6e82c4b80 1 -- start start 2021-11-19T08:40:42.306 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6e82c4b80 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d88a0 0x55f5961d8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.307 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6e82c4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d91b0 0x55f5961d9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.307 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6e82c4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5961174d0 con 0x55f5961d91b0 2021-11-19T08:40:42.307 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6e82c4b80 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f59612b3e0 con 0x55f5961d88a0 2021-11-19T08:40:42.307 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d91b0 0x55f5961d9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.307 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d91b0 0x55f5961d9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47106/0 (socket says 172.21.15.23:47106) 2021-11-19T08:40:42.308 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6cf7fe700 1 -- 172.21.15.23:0/2547690952 learned_addr learned my addr 172.21.15.23:0/2547690952 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:42.308 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6cffff700 1 --2- 172.21.15.23:0/2547690952 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d88a0 0x55f5961d8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.308 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6cf7fe700 1 -- 172.21.15.23:0/2547690952 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d88a0 msgr2=0x55f5961d8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-19T08:40:42.308 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6cf7fe700 1 --2- 172.21.15.23:0/2547690952 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d88a0 0x55f5961d8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6cf7fe700 1 -- 172.21.15.23:0/2547690952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5961a73c0 con 0x55f5961d91b0 2021-11-19T08:40:42.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6cffff700 1 --2- 172.21.15.23:0/2547690952 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d88a0 0x55f5961d8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:40:42.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6cf7fe700 1 --2- 172.21.15.23:0/2547690952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d91b0 0x55f5961d9b30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fc6c4014040 tx=0x7fc6c4016040).ready entity=mon.0 client_cookie=943fc1befa9521d2 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:42.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6b5ffb700 1 -- 172.21.15.23:0/2547690952 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c4014600 con 0x55f5961d91b0 2021-11-19T08:40:42.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6b5ffb700 1 -- 172.21.15.23:0/2547690952 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6c4049640 con 0x55f5961d91b0 2021-11-19T08:40:42.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.310+0000 7fc6b5ffb700 1 -- 172.21.15.23:0/2547690952 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c4075c10 con 0x55f5961d91b0 2021-11-19T08:40:42.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.311+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/2547690952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d91b0 msgr2=0x55f5961d9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.311+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/2547690952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d91b0 0x55f5961d9b30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fc6c4014040 tx=0x7fc6c4016040).stop 2021-11-19T08:40:42.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.311+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/2547690952 shutdown_connections 2021-11-19T08:40:42.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.311+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/2547690952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d91b0 0x55f5961d9b30 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.311+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/2547690952 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d88a0 0x55f5961d8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.311+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/2547690952 >> 172.21.15.23:0/2547690952 conn(0x55f5960461d0 msgr2=0x55f5961d3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:42.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.311+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/2547690952 shutdown_connections 2021-11-19T08:40:42.312 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.311+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/2547690952 wait complete. 2021-11-19T08:40:42.312 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.311+0000 7fc6e82c4b80 1 -- start start 2021-11-19T08:40:42.312 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.311+0000 7fc6e82c4b80 1 Processor -- start 2021-11-19T08:40:42.312 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6e82c4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5961e1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.312 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6e82c4b80 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 0x55f5961e2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.313 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6e82c4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f59612b520 con 0x55f5961d88a0 2021-11-19T08:40:42.313 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6e82c4b80 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f59612baa0 con 0x55f5961d91b0 2021-11-19T08:40:42.313 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6e82c4b80 10 client.0 ll_register_callbacks cb 0x55f5961988b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-19T08:40:42.313 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6cffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5961e1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.314 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6cffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5961e1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47108/0 (socket says 172.21.15.23:47108) 2021-11-19T08:40:42.314 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6cffff700 1 -- 172.21.15.23:0/655521754 learned_addr learned my addr 172.21.15.23:0/655521754 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:42.314 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6cf7fe700 1 --2- 172.21.15.23:0/655521754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 0x55f5961e2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.314 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6cffff700 1 -- 172.21.15.23:0/655521754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 msgr2=0x55f5961e2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-19T08:40:42.315 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6cffff700 1 --2- 172.21.15.23:0/655521754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 0x55f5961e2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.315 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6cffff700 1 -- 172.21.15.23:0/655521754 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d0108810 con 0x55f5961d88a0 2021-11-19T08:40:42.315 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6cf7fe700 1 --2- 172.21.15.23:0/655521754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 0x55f5961e2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:40:42.315 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6cffff700 1 --2- 172.21.15.23:0/655521754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5961e1b40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fc6c0014710 tx=0x7fc6c0016040).ready entity=mon.0 client_cookie=df66f591436fe634 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:42.316 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6aeffd700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2021-11-19T08:40:42.316 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6aeffd700 1 -- 172.21.15.23:0/655521754 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c0013760 con 0x55f5961d88a0 2021-11-19T08:40:42.316 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6aeffd700 1 -- 172.21.15.23:0/655521754 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fc6c00138c0 con 0x55f5961d88a0 2021-11-19T08:40:42.316 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.312+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/655521754 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f5961a6ce0 con 0x55f5961d88a0 2021-11-19T08:40:42.316 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.313+0000 7fc6aeffd700 1 -- 172.21.15.23:0/655521754 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6c0013ad0 con 0x55f5961d88a0 2021-11-19T08:40:42.317 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.313+0000 7fc6e82c4b80 10 client.14510 fetch_fsmap learned FSMap version 14 2021-11-19T08:40:42.317 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.313+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/655521754 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=3+}) v3 -- 0x55f59611a630 con 0x55f5961d88a0 2021-11-19T08:40:42.317 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.313+0000 7fc6aeffd700 1 -- 172.21.15.23:0/655521754 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x55f5961a6ce0 con 0x55f5961d88a0 2021-11-19T08:40:42.317 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.313+0000 7fc6aeffd700 1 -- 172.21.15.23:0/655521754 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 14) v1 ==== 1712+0+0 (secure 0 0 0) 0x7fc6c0021b50 con 0x55f5961d88a0 2021-11-19T08:40:42.318 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.313+0000 7fc6e82c4b80 10 client.14510 fetch_fsmap finished waiting for FSMap version 14 2021-11-19T08:40:42.318 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.313+0000 7fc6e82c4b80 10 client.14510 resolve_mds: resolved name 'cephfs.smithi023.fuvzkr' to daemon mds.cephfs.smithi023.fuvzkr 2021-11-19T08:40:42.318 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.313+0000 7fc6e82c4b80 20 client.14510 populate_metadata read hostname 'smithi023' 2021-11-19T08:40:42.318 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.313+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/655521754 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55f5961ea820 0x55f5961ecc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.319 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.313+0000 7fc6e82c4b80 4 client.14510 mds_command: new command op to 14434 tid=0[{"prefix": "get_command_descriptions"}] 2021-11-19T08:40:42.319 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.313+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/655521754 --> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f596121df0 con 0x55f5961ea820 2021-11-19T08:40:42.319 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.313+0000 7fc6cf7fe700 1 --2- 172.21.15.23:0/655521754 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55f5961ea820 0x55f5961ecc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.319 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.313+0000 7fc6aeffd700 10 client.14510 ms_handle_connect on v2:172.21.15.23:6826/530484646 2021-11-19T08:40:42.319 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.314+0000 7fc6cf7fe700 1 --2- 172.21.15.23:0/655521754 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55f5961ea820 0x55f5961ecc10 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=ab1862216f463ee0 server_cookie=c708adf379893f89 in_seq=0 out_seq=0 2021-11-19T08:40:42.320 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.314+0000 7fc6aeffd700 1 -- 172.21.15.23:0/655521754 <== mds.0 v2:172.21.15.23:6826/530484646 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x55f596121df0 con 0x55f5961ea820 2021-11-19T08:40:42.320 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.314+0000 7fc6aeffd700 10 client.14510 handle_command_reply: tid=0 2021-11-19T08:40:42.320 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.314+0000 7fc6e82c4b80 1 client.14510 shutdown 2021-11-19T08:40:42.320 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.314+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/655521754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 msgr2=0x55f5961e1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.321 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.314+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/655521754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5961e1b40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fc6c0014710 tx=0x7fc6c0016040).stop 2021-11-19T08:40:42.321 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.315+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/655521754 shutdown_connections 2021-11-19T08:40:42.321 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.315+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/655521754 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55f5961ea820 0x55f5961ecc10 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.321 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.315+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/655521754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5961e1b40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.321 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.315+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/655521754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 0x55f5961e2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.322 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.315+0000 7fc6aeffd700 0 client.14510 ms_handle_reset on v2:172.21.15.23:6826/530484646 2021-11-19T08:40:42.322 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.315+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/655521754 >> 172.21.15.23:0/655521754 conn(0x55f5960461d0 msgr2=0x55f5961d39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:42.322 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.316+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/655521754 shutdown_connections 2021-11-19T08:40:42.322 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.316+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/655521754 wait complete. 2021-11-19T08:40:42.323 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.316+0000 7fc6e82c4b80 20 client.14510 trim_cache size 0 max 16384 2021-11-19T08:40:42.329 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.332+0000 7fc6e82c4b80 1 Processor -- start 2021-11-19T08:40:42.329 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.332+0000 7fc6e82c4b80 1 -- start start 2021-11-19T08:40:42.329 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.332+0000 7fc6e82c4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5961d5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.330 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.332+0000 7fc6e82c4b80 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 0x55f5961d5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.330 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.332+0000 7fc6e82c4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5960f94a0 con 0x55f5961d88a0 2021-11-19T08:40:42.330 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.332+0000 7fc6e82c4b80 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f596121df0 con 0x55f5961d91b0 2021-11-19T08:40:42.330 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.332+0000 7fc6cffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5961d5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.331 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.333+0000 7fc6cffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5961d5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47114/0 (socket says 172.21.15.23:47114) 2021-11-19T08:40:42.331 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.333+0000 7fc6cffff700 1 -- 172.21.15.23:0/3395107421 learned_addr learned my addr 172.21.15.23:0/3395107421 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:42.331 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.333+0000 7fc6cf7fe700 1 --2- 172.21.15.23:0/3395107421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 0x55f5961d5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.331 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.333+0000 7fc6cffff700 1 -- 172.21.15.23:0/3395107421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 msgr2=0x55f5961d5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-19T08:40:42.331 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.333+0000 7fc6cffff700 1 --2- 172.21.15.23:0/3395107421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 0x55f5961d5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.332 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.333+0000 7fc6cffff700 1 -- 172.21.15.23:0/3395107421 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5961da5f0 con 0x55f5961d88a0 2021-11-19T08:40:42.332 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.333+0000 7fc6cffff700 1 --2- 172.21.15.23:0/3395107421 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5961d5520 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fc6c0022fd0 tx=0x7fc6c0009f90).ready entity=mon.0 client_cookie=a88833fee9fd3ec6 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:42.332 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.333+0000 7fc6b67fc700 1 -- 172.21.15.23:0/3395107421 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c002ce60 con 0x55f5961d88a0 2021-11-19T08:40:42.332 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.333+0000 7fc6b67fc700 1 -- 172.21.15.23:0/3395107421 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6c001a030 con 0x55f5961d88a0 2021-11-19T08:40:42.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.333+0000 7fc6b67fc700 1 -- 172.21.15.23:0/3395107421 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c00137e0 con 0x55f5961d88a0 2021-11-19T08:40:42.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.334+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/3395107421 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 msgr2=0x55f5961d5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.334+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/3395107421 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5961d5520 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fc6c0022fd0 tx=0x7fc6c0009f90).stop 2021-11-19T08:40:42.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.334+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/3395107421 shutdown_connections 2021-11-19T08:40:42.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.334+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/3395107421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 0x55f5961d5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.334+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/3395107421 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5961d5520 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.334+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/3395107421 >> 172.21.15.23:0/3395107421 conn(0x55f5960461d0 msgr2=0x55f5961d4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:42.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.334+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/3395107421 shutdown_connections 2021-11-19T08:40:42.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.334+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/3395107421 wait complete. 2021-11-19T08:40:42.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.334+0000 7fc6e82c4b80 1 -- start start 2021-11-19T08:40:42.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.334+0000 7fc6e82c4b80 1 Processor -- start 2021-11-19T08:40:42.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6e82c4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5962368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6e82c4b80 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 0x55f596236dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6e82c4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f596237370 con 0x55f5961d88a0 2021-11-19T08:40:42.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6e82c4b80 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5962374b0 con 0x55f5961d91b0 2021-11-19T08:40:42.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6e82c4b80 10 client.0 ll_register_callbacks cb 0x55f5961988b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-19T08:40:42.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6cffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5962368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6cffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5962368d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47118/0 (socket says 172.21.15.23:47118) 2021-11-19T08:40:42.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6cffff700 1 -- 172.21.15.23:0/1474485354 learned_addr learned my addr 172.21.15.23:0/1474485354 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:42.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6cf7fe700 1 --2- 172.21.15.23:0/1474485354 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 0x55f596236dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6cffff700 1 -- 172.21.15.23:0/1474485354 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 msgr2=0x55f596236dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-19T08:40:42.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6cffff700 1 --2- 172.21.15.23:0/1474485354 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 0x55f596236dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6cffff700 1 -- 172.21.15.23:0/1474485354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f59611a630 con 0x55f5961d88a0 2021-11-19T08:40:42.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6cffff700 1 --2- 172.21.15.23:0/1474485354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5962368d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fc6c00097d0 tx=0x7fc6c0009f90).ready entity=mon.0 client_cookie=17ac0229dcd47813 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:42.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6aeffd700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2021-11-19T08:40:42.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6aeffd700 1 -- 172.21.15.23:0/1474485354 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c002ba40 con 0x55f5961d88a0 2021-11-19T08:40:42.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6aeffd700 1 -- 172.21.15.23:0/1474485354 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fc6c0030070 con 0x55f5961d88a0 2021-11-19T08:40:42.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.335+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/1474485354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f5961d2fc0 con 0x55f5961d88a0 2021-11-19T08:40:42.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.336+0000 7fc6aeffd700 1 -- 172.21.15.23:0/1474485354 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6c002bba0 con 0x55f5961d88a0 2021-11-19T08:40:42.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.336+0000 7fc6e82c4b80 10 client.14514 fetch_fsmap learned FSMap version 14 2021-11-19T08:40:42.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.336+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/1474485354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=3+}) v3 -- 0x7fc6d010f720 con 0x55f5961d88a0 2021-11-19T08:40:42.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.336+0000 7fc6aeffd700 1 -- 172.21.15.23:0/1474485354 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x55f5961d2fc0 con 0x55f5961d88a0 2021-11-19T08:40:42.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.336+0000 7fc6aeffd700 1 -- 172.21.15.23:0/1474485354 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 14) v1 ==== 1712+0+0 (secure 0 0 0) 0x7fc6c0021430 con 0x55f5961d88a0 2021-11-19T08:40:42.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.336+0000 7fc6e82c4b80 10 client.14514 fetch_fsmap finished waiting for FSMap version 14 2021-11-19T08:40:42.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.336+0000 7fc6e82c4b80 10 client.14514 resolve_mds: resolved name 'cephfs.smithi023.fuvzkr' to daemon mds.cephfs.smithi023.fuvzkr 2021-11-19T08:40:42.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.336+0000 7fc6e82c4b80 20 client.14514 populate_metadata read hostname 'smithi023' 2021-11-19T08:40:42.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.336+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/1474485354 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55f59623cc50 0x55f59623f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.336+0000 7fc6e82c4b80 4 client.14514 mds_command: new command op to 14434 tid=0[{"prefix": "status"}] 2021-11-19T08:40:42.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.336+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/1474485354 --> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f5961190c0 con 0x55f59623cc50 2021-11-19T08:40:42.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.336+0000 7fc6cf7fe700 1 --2- 172.21.15.23:0/1474485354 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55f59623cc50 0x55f59623f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.336+0000 7fc6aeffd700 10 client.14514 ms_handle_connect on v2:172.21.15.23:6826/530484646 2021-11-19T08:40:42.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6cf7fe700 1 --2- 172.21.15.23:0/1474485354 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55f59623cc50 0x55f59623f040 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=faf81b5281b6f0ea server_cookie=ee460b46b8081c6c in_seq=0 out_seq=0 2021-11-19T08:40:42.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6aeffd700 1 -- 172.21.15.23:0/1474485354 <== mds.0 v2:172.21.15.23:6826/530484646 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55f5961190c0 con 0x55f59623cc50 2021-11-19T08:40:42.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6aeffd700 10 client.14514 handle_command_reply: tid=0 2021-11-19T08:40:42.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6e82c4b80 1 client.14514 shutdown 2021-11-19T08:40:42.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/1474485354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 msgr2=0x55f5962368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/1474485354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5962368d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fc6c00097d0 tx=0x7fc6c0009f90).stop 2021-11-19T08:40:42.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/1474485354 shutdown_connections 2021-11-19T08:40:42.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/1474485354 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55f59623cc50 0x55f59623f040 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.344 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/1474485354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5961d88a0 0x55f5962368d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.344 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6e82c4b80 1 --2- 172.21.15.23:0/1474485354 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f5961d91b0 0x55f596236dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.344 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/1474485354 >> 172.21.15.23:0/1474485354 conn(0x55f5960461d0 msgr2=0x55f5961d5630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:42.344 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6aeffd700 0 client.14514 ms_handle_reset on v2:172.21.15.23:6826/530484646 2021-11-19T08:40:42.345 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/1474485354 shutdown_connections 2021-11-19T08:40:42.345 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6e82c4b80 1 -- 172.21.15.23:0/1474485354 wait complete. 2021-11-19T08:40:42.345 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.337+0000 7fc6e82c4b80 20 client.14514 trim_cache size 0 max 16384 2021-11-19T08:40:42.346 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2021-11-19T08:40:42.346 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "908b41ee-4913-11ec-8c2c-001a4aab830c", 2021-11-19T08:40:42.346 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2021-11-19T08:40:42.346 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 14434, 2021-11-19T08:40:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2021-11-19T08:40:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2021-11-19T08:40:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2021-11-19T08:40:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 17.111725567000001, 2021-11-19T08:40:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 14, 2021-11-19T08:40:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 41, 2021-11-19T08:40:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 41, 2021-11-19T08:40:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 17.950950735999999 2021-11-19T08:40:42.348 INFO:teuthology.orchestra.run.smithi023.stdout:} 2021-11-19T08:40:42.350 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.338+0000 7fc6d54b7700 1 -- 172.21.15.23:0/2146037625 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc6b805c2b0 msgr2=0x7fc6b805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.350 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.338+0000 7fc6d54b7700 1 --2- 172.21.15.23:0/2146037625 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc6b805c2b0 0x7fc6b805e770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fc6c0009da0 tx=0x7fc6c000b040).stop 2021-11-19T08:40:42.351 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.338+0000 7fc6d54b7700 1 -- 172.21.15.23:0/2146037625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d00f6690 msgr2=0x7fc6d0107280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.351 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.338+0000 7fc6d54b7700 1 --2- 172.21.15.23:0/2146037625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d00f6690 0x7fc6d0107280 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fc6c4000c00 tx=0x7fc6c400b040).stop 2021-11-19T08:40:42.351 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.338+0000 7fc6d54b7700 1 -- 172.21.15.23:0/2146037625 shutdown_connections 2021-11-19T08:40:42.351 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.338+0000 7fc6d54b7700 1 --2- 172.21.15.23:0/2146037625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6d00f53f0 0x7fc6d0106d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.352 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.338+0000 7fc6d54b7700 1 --2- 172.21.15.23:0/2146037625 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc6b805c2b0 0x7fc6b805e770 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.352 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.338+0000 7fc6d54b7700 1 --2- 172.21.15.23:0/2146037625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d00f6690 0x7fc6d0107280 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.352 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.338+0000 7fc6d54b7700 1 -- 172.21.15.23:0/2146037625 >> 172.21.15.23:0/2146037625 conn(0x7fc6d00f0b30 msgr2=0x7fc6d005fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:42.352 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.338+0000 7fc6d54b7700 1 -- 172.21.15.23:0/2146037625 shutdown_connections 2021-11-19T08:40:42.352 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.338+0000 7fc6d54b7700 1 -- 172.21.15.23:0/2146037625 wait complete. 2021-11-19T08:40:42.353 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.cephfs.smithi134.opxllb status 2021-11-19T08:40:42.476 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.488+0000 7f5e11d7d700 1 -- 172.21.15.23:0/3347246657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c0f7b40 msgr2=0x7f5e0c0f7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.477 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.488+0000 7f5e11d7d700 1 --2- 172.21.15.23:0/3347246657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c0f7b40 0x7f5e0c0f7fa0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f5e00004660 tx=0x7f5e00009b30).stop 2021-11-19T08:40:42.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.488+0000 7f5e11d7d700 1 -- 172.21.15.23:0/3347246657 shutdown_connections 2021-11-19T08:40:42.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.488+0000 7f5e11d7d700 1 --2- 172.21.15.23:0/3347246657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c0f7b40 0x7f5e0c0f7fa0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.488+0000 7f5e11d7d700 1 --2- 172.21.15.23:0/3347246657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e0c0f68a0 0x7f5e0c0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.488+0000 7f5e11d7d700 1 -- 172.21.15.23:0/3347246657 >> 172.21.15.23:0/3347246657 conn(0x7f5e0c0f1fc0 msgr2=0x7f5e0c0f4420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:42.479 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.488+0000 7f5e11d7d700 1 -- 172.21.15.23:0/3347246657 shutdown_connections 2021-11-19T08:40:42.479 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.488+0000 7f5e11d7d700 1 -- 172.21.15.23:0/3347246657 wait complete. 2021-11-19T08:40:42.479 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.489+0000 7f5e11d7d700 1 Processor -- start 2021-11-19T08:40:42.479 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.489+0000 7f5e11d7d700 1 -- start start 2021-11-19T08:40:42.480 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.489+0000 7f5e11d7d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c0f68a0 0x7f5e0c107060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.480 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.489+0000 7f5e11d7d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e0c0f7b40 0x7f5e0c107550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.481 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.489+0000 7f5e11d7d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c107b00 con 0x7f5e0c0f68a0 2021-11-19T08:40:42.481 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.489+0000 7f5e11d7d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c107c40 con 0x7f5e0c0f7b40 2021-11-19T08:40:42.481 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.489+0000 7f5e0b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c0f68a0 0x7f5e0c107060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.482 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.489+0000 7f5e0b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c0f68a0 0x7f5e0c107060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47124/0 (socket says 172.21.15.23:47124) 2021-11-19T08:40:42.482 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.489+0000 7f5e0b7fe700 1 -- 172.21.15.23:0/2134173344 learned_addr learned my addr 172.21.15.23:0/2134173344 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:42.482 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.490+0000 7f5e0affd700 1 --2- 172.21.15.23:0/2134173344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e0c0f7b40 0x7f5e0c107550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.490+0000 7f5e0b7fe700 1 -- 172.21.15.23:0/2134173344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e0c0f7b40 msgr2=0x7f5e0c107550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.490+0000 7f5e0b7fe700 1 --2- 172.21.15.23:0/2134173344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e0c0f7b40 0x7f5e0c107550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.490+0000 7f5e0b7fe700 1 -- 172.21.15.23:0/2134173344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e00005040 con 0x7f5e0c0f68a0 2021-11-19T08:40:42.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.490+0000 7f5e0affd700 1 --2- 172.21.15.23:0/2134173344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e0c0f7b40 0x7f5e0c107550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:40:42.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.490+0000 7f5e0b7fe700 1 --2- 172.21.15.23:0/2134173344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c0f68a0 0x7f5e0c107060 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f5dfc002700 tx=0x7f5dfc009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:42.484 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.490+0000 7f5df3fff700 1 -- 172.21.15.23:0/2134173344 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dfc009320 con 0x7f5e0c0f68a0 2021-11-19T08:40:42.484 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.490+0000 7f5df3fff700 1 -- 172.21.15.23:0/2134173344 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5dfc00f040 con 0x7f5e0c0f68a0 2021-11-19T08:40:42.484 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.490+0000 7f5df3fff700 1 -- 172.21.15.23:0/2134173344 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dfc014900 con 0x7f5e0c0f68a0 2021-11-19T08:40:42.484 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.490+0000 7f5e11d7d700 1 -- 172.21.15.23:0/2134173344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0c10c6e0 con 0x7f5e0c0f68a0 2021-11-19T08:40:42.485 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.490+0000 7f5e11d7d700 1 -- 172.21.15.23:0/2134173344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c10cca0 con 0x7f5e0c0f68a0 2021-11-19T08:40:42.485 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.492+0000 7f5df3fff700 1 -- 172.21.15.23:0/2134173344 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5dfc009480 con 0x7f5e0c0f68a0 2021-11-19T08:40:42.492 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.492+0000 7f5df3fff700 1 --2- 172.21.15.23:0/2134173344 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5df405c2a0 0x7f5df405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.492 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.492+0000 7f5df3fff700 1 -- 172.21.15.23:0/2134173344 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f5dfc076180 con 0x7f5e0c0f68a0 2021-11-19T08:40:42.493 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.492+0000 7f5e0affd700 1 --2- 172.21.15.23:0/2134173344 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5df405c2a0 0x7f5df405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.493 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.493+0000 7f5e0affd700 1 --2- 172.21.15.23:0/2134173344 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5df405c2a0 0x7f5df405e760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f5e00009d60 tx=0x7f5e00009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:42.493 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.495+0000 7f5e23928b80 1 Processor -- start 2021-11-19T08:40:42.493 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.495+0000 7f5e23928b80 1 -- start start 2021-11-19T08:40:42.494 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.495+0000 7f5e23928b80 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40e271d0 0x55fe40fb9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.494 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.495+0000 7f5e23928b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40fb9fe0 0x55fe40fbad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.494 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.495+0000 7f5e23928b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe40ef84d0 con 0x55fe40fb9fe0 2021-11-19T08:40:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.495+0000 7f5e23928b80 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe40f0c3e0 con 0x55fe40e271d0 2021-11-19T08:40:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.495+0000 7f5e0affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40fb9fe0 0x55fe40fbad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.495+0000 7f5e0affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40fb9fe0 0x55fe40fbad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47130/0 (socket says 172.21.15.23:47130) 2021-11-19T08:40:42.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.495+0000 7f5e0affd700 1 -- 172.21.15.23:0/426689721 learned_addr learned my addr 172.21.15.23:0/426689721 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.495+0000 7f5e0b7fe700 1 --2- 172.21.15.23:0/426689721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40e271d0 0x55fe40fb9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.495+0000 7f5e0affd700 1 -- 172.21.15.23:0/426689721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40e271d0 msgr2=0x55fe40fb9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-19T08:40:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.495+0000 7f5e0affd700 1 --2- 172.21.15.23:0/426689721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40e271d0 0x55fe40fb9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.496+0000 7f5e0affd700 1 -- 172.21.15.23:0/426689721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe40f883c0 con 0x55fe40fb9fe0 2021-11-19T08:40:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.496+0000 7f5e0affd700 1 --2- 172.21.15.23:0/426689721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40fb9fe0 0x55fe40fbad40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f5e000043d0 tx=0x7f5e0001b000).ready entity=mon.0 client_cookie=6ec5255d671d5f65 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:42.497 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.496+0000 7f5df17fa700 1 -- 172.21.15.23:0/426689721 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e00019070 con 0x55fe40fb9fe0 2021-11-19T08:40:42.497 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.496+0000 7f5df17fa700 1 -- 172.21.15.23:0/426689721 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e00013e50 con 0x55fe40fb9fe0 2021-11-19T08:40:42.497 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.496+0000 7f5df17fa700 1 -- 172.21.15.23:0/426689721 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e0000e040 con 0x55fe40fb9fe0 2021-11-19T08:40:42.497 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.497+0000 7f5e23928b80 1 -- 172.21.15.23:0/426689721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40fb9fe0 msgr2=0x55fe40fbad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.498 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.497+0000 7f5e23928b80 1 --2- 172.21.15.23:0/426689721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40fb9fe0 0x55fe40fbad40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f5e000043d0 tx=0x7f5e0001b000).stop 2021-11-19T08:40:42.498 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.497+0000 7f5e23928b80 1 -- 172.21.15.23:0/426689721 shutdown_connections 2021-11-19T08:40:42.498 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.497+0000 7f5e23928b80 1 --2- 172.21.15.23:0/426689721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40fb9fe0 0x55fe40fbad40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.498 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.497+0000 7f5e23928b80 1 --2- 172.21.15.23:0/426689721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40e271d0 0x55fe40fb9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.499 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.497+0000 7f5e23928b80 1 -- 172.21.15.23:0/426689721 >> 172.21.15.23:0/426689721 conn(0x55fe40f7f9b0 msgr2=0x55fe40fb4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:42.499 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.497+0000 7f5e23928b80 1 -- 172.21.15.23:0/426689721 shutdown_connections 2021-11-19T08:40:42.499 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.497+0000 7f5e23928b80 1 -- 172.21.15.23:0/426689721 wait complete. 2021-11-19T08:40:42.500 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.497+0000 7f5e23928b80 1 -- start start 2021-11-19T08:40:42.500 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.497+0000 7f5e23928b80 1 Processor -- start 2021-11-19T08:40:42.500 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.497+0000 7f5e23928b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40e271d0 0x55fe40fc2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.501 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.497+0000 7f5e23928b80 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40fb9fe0 0x55fe40fc2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.501 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.497+0000 7f5e23928b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe40f0c520 con 0x55fe40e271d0 2021-11-19T08:40:42.501 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.498+0000 7f5e23928b80 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe40f0caa0 con 0x55fe40fb9fe0 2021-11-19T08:40:42.502 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.498+0000 7f5e23928b80 10 client.0 ll_register_callbacks cb 0x55fe40f79700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-19T08:40:42.502 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.498+0000 7f5e0b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40e271d0 0x55fe40fc2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.502 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.498+0000 7f5e0b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40e271d0 0x55fe40fc2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47136/0 (socket says 172.21.15.23:47136) 2021-11-19T08:40:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.498+0000 7f5e0b7fe700 1 -- 172.21.15.23:0/2819787631 learned_addr learned my addr 172.21.15.23:0/2819787631 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.498+0000 7f5e0affd700 1 --2- 172.21.15.23:0/2819787631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40fb9fe0 0x55fe40fc2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.498+0000 7f5e0b7fe700 1 -- 172.21.15.23:0/2819787631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40fb9fe0 msgr2=0x55fe40fc2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-19T08:40:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.498+0000 7f5e0b7fe700 1 --2- 172.21.15.23:0/2819787631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40fb9fe0 0x55fe40fc2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.498+0000 7f5e0b7fe700 1 -- 172.21.15.23:0/2819787631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c0f8620 con 0x55fe40e271d0 2021-11-19T08:40:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.498+0000 7f5e0b7fe700 1 --2- 172.21.15.23:0/2819787631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40e271d0 0x55fe40fc2840 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f5dfc007e20 tx=0x7f5dfc007d70).ready entity=mon.0 client_cookie=1e5d1f33ede301d4 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.498+0000 7f5dea7fc700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2021-11-19T08:40:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.498+0000 7f5dea7fc700 1 -- 172.21.15.23:0/2819787631 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dfc04cb20 con 0x55fe40e271d0 2021-11-19T08:40:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.498+0000 7f5dea7fc700 1 -- 172.21.15.23:0/2819787631 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f5dfc048b80 con 0x55fe40e271d0 2021-11-19T08:40:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.498+0000 7f5e23928b80 1 -- 172.21.15.23:0/2819787631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe40f87ce0 con 0x55fe40e271d0 2021-11-19T08:40:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.500+0000 7f5dea7fc700 1 -- 172.21.15.23:0/2819787631 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5dfc0077b0 con 0x55fe40e271d0 2021-11-19T08:40:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.500+0000 7f5e23928b80 10 client.14522 fetch_fsmap learned FSMap version 14 2021-11-19T08:40:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.500+0000 7f5e23928b80 1 -- 172.21.15.23:0/2819787631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=3+}) v3 -- 0x55fe40efb630 con 0x55fe40e271d0 2021-11-19T08:40:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.500+0000 7f5dea7fc700 1 -- 172.21.15.23:0/2819787631 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x55fe40f87ce0 con 0x55fe40e271d0 2021-11-19T08:40:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.501+0000 7f5dea7fc700 1 -- 172.21.15.23:0/2819787631 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 14) v1 ==== 1712+0+0 (secure 0 0 0) 0x7f5dfc04a270 con 0x55fe40e271d0 2021-11-19T08:40:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.501+0000 7f5e23928b80 10 client.14522 fetch_fsmap finished waiting for FSMap version 14 2021-11-19T08:40:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.501+0000 7f5e23928b80 10 client.14522 resolve_mds: resolved name 'cephfs.smithi134.opxllb' to daemon mds.cephfs.smithi134.opxllb 2021-11-19T08:40:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.501+0000 7f5e23928b80 20 client.14522 populate_metadata read hostname 'smithi023' 2021-11-19T08:40:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.501+0000 7f5e23928b80 1 --2- 172.21.15.23:0/2819787631 >> [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987] conn(0x55fe40fcb510 0x55fe40fcd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.501+0000 7f5e23928b80 4 client.14522 mds_command: new command op to 24283 tid=0[{"prefix": "get_command_descriptions"}] 2021-11-19T08:40:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.501+0000 7f5e23928b80 1 -- 172.21.15.23:0/2819787631 --> [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe40f02df0 con 0x55fe40fcb510 2021-11-19T08:40:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.501+0000 7f5e0affd700 1 --2- 172.21.15.23:0/2819787631 >> [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987] conn(0x55fe40fcb510 0x55fe40fcd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.502+0000 7f5dea7fc700 10 client.14522 ms_handle_connect on v2:172.21.15.134:6826/3905671987 2021-11-19T08:40:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.502+0000 7f5e0affd700 1 --2- 172.21.15.23:0/2819787631 >> [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987] conn(0x55fe40fcb510 0x55fe40fcd900 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.1 client_cookie=4ffe1fb529b021f8 server_cookie=e18b1c2c61a46234 in_seq=0 out_seq=0 2021-11-19T08:40:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.502+0000 7f5dea7fc700 1 -- 172.21.15.23:0/2819787631 <== mds.1 v2:172.21.15.134:6826/3905671987 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x55fe40f02df0 con 0x55fe40fcb510 2021-11-19T08:40:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.502+0000 7f5dea7fc700 10 client.14522 handle_command_reply: tid=0 2021-11-19T08:40:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.502+0000 7f5e23928b80 1 client.14522 shutdown 2021-11-19T08:40:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.502+0000 7f5e23928b80 1 -- 172.21.15.23:0/2819787631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40e271d0 msgr2=0x55fe40fc2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.502+0000 7f5e23928b80 1 --2- 172.21.15.23:0/2819787631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40e271d0 0x55fe40fc2840 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f5dfc007e20 tx=0x7f5dfc007d70).stop 2021-11-19T08:40:42.511 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.502+0000 7f5e23928b80 1 -- 172.21.15.23:0/2819787631 shutdown_connections 2021-11-19T08:40:42.511 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.503+0000 7f5e23928b80 1 --2- 172.21.15.23:0/2819787631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40e271d0 0x55fe40fc2840 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.511 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.503+0000 7f5e23928b80 1 --2- 172.21.15.23:0/2819787631 >> [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987] conn(0x55fe40fcb510 0x55fe40fcd900 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.512 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.503+0000 7f5e23928b80 1 --2- 172.21.15.23:0/2819787631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40fb9fe0 0x55fe40fc2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.512 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.503+0000 7f5dea7fc700 0 client.14522 ms_handle_reset on v2:172.21.15.134:6826/3905671987 2021-11-19T08:40:42.513 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.503+0000 7f5e23928b80 1 -- 172.21.15.23:0/2819787631 >> 172.21.15.23:0/2819787631 conn(0x55fe40f7f9b0 msgr2=0x55fe40fbb270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:42.513 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.503+0000 7f5e23928b80 1 -- 172.21.15.23:0/2819787631 shutdown_connections 2021-11-19T08:40:42.513 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.503+0000 7f5e23928b80 1 -- 172.21.15.23:0/2819787631 wait complete. 2021-11-19T08:40:42.514 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.503+0000 7f5e23928b80 20 client.14522 trim_cache size 0 max 16384 2021-11-19T08:40:42.523 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.519+0000 7f5e23928b80 1 Processor -- start 2021-11-19T08:40:42.523 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.519+0000 7f5e23928b80 1 -- start start 2021-11-19T08:40:42.523 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.519+0000 7f5e23928b80 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40e271d0 0x55fe40fb29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.524 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.519+0000 7f5e23928b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40fb9fe0 0x55fe40fb2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.524 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.519+0000 7f5e23928b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe40eda4a0 con 0x55fe40fb9fe0 2021-11-19T08:40:42.525 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.519+0000 7f5e23928b80 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe40f02df0 con 0x55fe40e271d0 2021-11-19T08:40:42.525 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.519+0000 7f5e0affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40fb9fe0 0x55fe40fb2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.519+0000 7f5e0affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40fb9fe0 0x55fe40fb2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47140/0 (socket says 172.21.15.23:47140) 2021-11-19T08:40:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.519+0000 7f5e0affd700 1 -- 172.21.15.23:0/330899774 learned_addr learned my addr 172.21.15.23:0/330899774 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.520+0000 7f5e0b7fe700 1 --2- 172.21.15.23:0/330899774 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40e271d0 0x55fe40fb29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.520+0000 7f5e0affd700 1 -- 172.21.15.23:0/330899774 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40e271d0 msgr2=0x55fe40fb29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-19T08:40:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.520+0000 7f5e0affd700 1 --2- 172.21.15.23:0/330899774 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40e271d0 0x55fe40fb29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.520+0000 7f5e0affd700 1 -- 172.21.15.23:0/330899774 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe40fb62f0 con 0x55fe40fb9fe0 2021-11-19T08:40:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.520+0000 7f5e0affd700 1 --2- 172.21.15.23:0/330899774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40fb9fe0 0x55fe40fb2ed0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f5e00004630 tx=0x7f5e00022040).ready entity=mon.0 client_cookie=b9587c85db4f5c0b server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.520+0000 7f5df1ffb700 1 -- 172.21.15.23:0/330899774 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e0000e030 con 0x55fe40fb9fe0 2021-11-19T08:40:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.520+0000 7f5df1ffb700 1 -- 172.21.15.23:0/330899774 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e0001dae0 con 0x55fe40fb9fe0 2021-11-19T08:40:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.520+0000 7f5df1ffb700 1 -- 172.21.15.23:0/330899774 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e0002f070 con 0x55fe40fb9fe0 2021-11-19T08:40:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 1 -- 172.21.15.23:0/330899774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40fb9fe0 msgr2=0x55fe40fb2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 1 --2- 172.21.15.23:0/330899774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40fb9fe0 0x55fe40fb2ed0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f5e00004630 tx=0x7f5e00022040).stop 2021-11-19T08:40:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 1 -- 172.21.15.23:0/330899774 shutdown_connections 2021-11-19T08:40:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 1 --2- 172.21.15.23:0/330899774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40fb9fe0 0x55fe40fb2ed0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.530 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 1 --2- 172.21.15.23:0/330899774 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40e271d0 0x55fe40fb29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.530 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 1 -- 172.21.15.23:0/330899774 >> 172.21.15.23:0/330899774 conn(0x55fe40f7f9b0 msgr2=0x55fe40fc15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:42.530 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 1 -- 172.21.15.23:0/330899774 shutdown_connections 2021-11-19T08:40:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 1 -- 172.21.15.23:0/330899774 wait complete. 2021-11-19T08:40:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 1 -- start start 2021-11-19T08:40:42.532 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 1 Processor -- start 2021-11-19T08:40:42.532 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40e271d0 0x55fe41017850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.532 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40fb9fe0 0x55fe41017d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.532 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe410182f0 con 0x55fe40e271d0 2021-11-19T08:40:42.533 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe41018430 con 0x55fe40fb9fe0 2021-11-19T08:40:42.533 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e23928b80 10 client.0 ll_register_callbacks cb 0x55fe40f79700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-19T08:40:42.533 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e0b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40e271d0 0x55fe41017850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.533 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e0b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40e271d0 0x55fe41017850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47144/0 (socket says 172.21.15.23:47144) 2021-11-19T08:40:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.521+0000 7f5e0b7fe700 1 -- 172.21.15.23:0/3799025641 learned_addr learned my addr 172.21.15.23:0/3799025641 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5e0affd700 1 --2- 172.21.15.23:0/3799025641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40fb9fe0 0x55fe41017d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5e0b7fe700 1 -- 172.21.15.23:0/3799025641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40fb9fe0 msgr2=0x55fe41017d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-19T08:40:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5e0b7fe700 1 --2- 172.21.15.23:0/3799025641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40fb9fe0 0x55fe41017d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.535 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5e0b7fe700 1 -- 172.21.15.23:0/3799025641 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe40f883c0 con 0x55fe40e271d0 2021-11-19T08:40:42.535 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5e0b7fe700 1 --2- 172.21.15.23:0/3799025641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40e271d0 0x55fe41017850 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f5dfc002f20 tx=0x7f5dfc05a040).ready entity=mon.0 client_cookie=88113482a2da4c56 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:42.535 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5dea7fc700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2021-11-19T08:40:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5dea7fc700 1 -- 172.21.15.23:0/3799025641 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dfc01ae40 con 0x55fe40e271d0 2021-11-19T08:40:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5e23928b80 1 -- 172.21.15.23:0/3799025641 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe40fb4220 con 0x55fe40e271d0 2021-11-19T08:40:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5dea7fc700 1 -- 172.21.15.23:0/3799025641 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f5dfc04cb20 con 0x55fe40e271d0 2021-11-19T08:40:42.537 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5dea7fc700 1 -- 172.21.15.23:0/3799025641 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5dfc048b80 con 0x55fe40e271d0 2021-11-19T08:40:42.537 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5e23928b80 10 client.14526 fetch_fsmap learned FSMap version 14 2021-11-19T08:40:42.537 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5e23928b80 1 -- 172.21.15.23:0/3799025641 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=3+}) v3 -- 0x7f5e0c10f900 con 0x55fe40e271d0 2021-11-19T08:40:42.538 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5dea7fc700 1 -- 172.21.15.23:0/3799025641 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x55fe40fb4220 con 0x55fe40e271d0 2021-11-19T08:40:42.538 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5dea7fc700 1 -- 172.21.15.23:0/3799025641 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 14) v1 ==== 1712+0+0 (secure 0 0 0) 0x7f5dfc0742f0 con 0x55fe40e271d0 2021-11-19T08:40:42.538 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5e23928b80 10 client.14526 fetch_fsmap finished waiting for FSMap version 14 2021-11-19T08:40:42.538 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5e23928b80 10 client.14526 resolve_mds: resolved name 'cephfs.smithi134.opxllb' to daemon mds.cephfs.smithi134.opxllb 2021-11-19T08:40:42.539 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5e23928b80 20 client.14526 populate_metadata read hostname 'smithi023' 2021-11-19T08:40:42.539 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5e23928b80 1 --2- 172.21.15.23:0/3799025641 >> [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987] conn(0x55fe4101db20 0x55fe4101ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:42.539 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5e23928b80 4 client.14526 mds_command: new command op to 24283 tid=0[{"prefix": "status"}] 2021-11-19T08:40:42.540 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.522+0000 7f5e23928b80 1 -- 172.21.15.23:0/3799025641 --> [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fe40efa0c0 con 0x55fe4101db20 2021-11-19T08:40:42.540 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5e0affd700 1 --2- 172.21.15.23:0/3799025641 >> [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987] conn(0x55fe4101db20 0x55fe4101ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:42.540 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5dea7fc700 10 client.14526 ms_handle_connect on v2:172.21.15.134:6826/3905671987 2021-11-19T08:40:42.541 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5e0affd700 1 --2- 172.21.15.23:0/3799025641 >> [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987] conn(0x55fe4101db20 0x55fe4101ff30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.1 client_cookie=a2609d28e05bdaf4 server_cookie=87aaa28b088273e2 in_seq=0 out_seq=0 2021-11-19T08:40:42.541 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5dea7fc700 1 -- 172.21.15.23:0/3799025641 <== mds.1 v2:172.21.15.134:6826/3905671987 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55fe40efa0c0 con 0x55fe4101db20 2021-11-19T08:40:42.541 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5dea7fc700 10 client.14526 handle_command_reply: tid=0 2021-11-19T08:40:42.541 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5e23928b80 1 client.14526 shutdown 2021-11-19T08:40:42.541 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5e23928b80 1 -- 172.21.15.23:0/3799025641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40e271d0 msgr2=0x55fe41017850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.542 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5e23928b80 1 --2- 172.21.15.23:0/3799025641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40e271d0 0x55fe41017850 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f5dfc002f20 tx=0x7f5dfc05a040).stop 2021-11-19T08:40:42.542 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5e23928b80 1 -- 172.21.15.23:0/3799025641 shutdown_connections 2021-11-19T08:40:42.542 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5e23928b80 1 --2- 172.21.15.23:0/3799025641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe40e271d0 0x55fe41017850 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.542 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5e23928b80 1 --2- 172.21.15.23:0/3799025641 >> [v2:172.21.15.134:6826/3905671987,v1:172.21.15.134:6827/3905671987] conn(0x55fe4101db20 0x55fe4101ff30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5e23928b80 1 --2- 172.21.15.23:0/3799025641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fe40fb9fe0 0x55fe41017d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5dea7fc700 0 client.14526 ms_handle_reset on v2:172.21.15.134:6826/3905671987 2021-11-19T08:40:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5e23928b80 1 -- 172.21.15.23:0/3799025641 >> 172.21.15.23:0/3799025641 conn(0x55fe40f7f9b0 msgr2=0x55fe40fb29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5e23928b80 1 -- 172.21.15.23:0/3799025641 shutdown_connections 2021-11-19T08:40:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5e23928b80 1 -- 172.21.15.23:0/3799025641 wait complete. 2021-11-19T08:40:42.544 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.523+0000 7f5e23928b80 20 client.14526 trim_cache size 0 max 16384 2021-11-19T08:40:42.544 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2021-11-19T08:40:42.545 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "908b41ee-4913-11ec-8c2c-001a4aab830c", 2021-11-19T08:40:42.545 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 1, 2021-11-19T08:40:42.545 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24283, 2021-11-19T08:40:42.545 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2021-11-19T08:40:42.546 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2021-11-19T08:40:42.546 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2021-11-19T08:40:42.546 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 11.738755085999999, 2021-11-19T08:40:42.546 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 14, 2021-11-19T08:40:42.546 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 41, 2021-11-19T08:40:42.547 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 41, 2021-11-19T08:40:42.547 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 11.890471081999999 2021-11-19T08:40:42.547 INFO:teuthology.orchestra.run.smithi023.stdout:} 2021-11-19T08:40:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.524+0000 7f5e11d7d700 1 -- 172.21.15.23:0/2134173344 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5df405c2a0 msgr2=0x7f5df405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.524+0000 7f5e11d7d700 1 --2- 172.21.15.23:0/2134173344 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5df405c2a0 0x7f5df405e760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f5e00009d60 tx=0x7f5e00009b30).stop 2021-11-19T08:40:42.550 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.524+0000 7f5e11d7d700 1 -- 172.21.15.23:0/2134173344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c0f68a0 msgr2=0x7f5e0c107060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:42.550 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.524+0000 7f5e11d7d700 1 --2- 172.21.15.23:0/2134173344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c0f68a0 0x7f5e0c107060 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f5dfc002700 tx=0x7f5dfc009d80).stop 2021-11-19T08:40:42.550 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.524+0000 7f5e11d7d700 1 -- 172.21.15.23:0/2134173344 shutdown_connections 2021-11-19T08:40:42.551 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.524+0000 7f5e11d7d700 1 --2- 172.21.15.23:0/2134173344 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5df405c2a0 0x7f5df405e760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.551 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.524+0000 7f5e11d7d700 1 --2- 172.21.15.23:0/2134173344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c0f68a0 0x7f5e0c107060 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.552 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.524+0000 7f5e11d7d700 1 --2- 172.21.15.23:0/2134173344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e0c0f7b40 0x7f5e0c107550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:42.552 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.524+0000 7f5e11d7d700 1 -- 172.21.15.23:0/2134173344 >> 172.21.15.23:0/2134173344 conn(0x7f5e0c0f1fc0 msgr2=0x7f5e0c0fb590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:42.553 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.524+0000 7f5e11d7d700 1 -- 172.21.15.23:0/2134173344 shutdown_connections 2021-11-19T08:40:42.553 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:42.524+0000 7f5e11d7d700 1 -- 172.21.15.23:0/2134173344 wait complete. 2021-11-19T08:40:42.553 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:42 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:40:42.554 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:42 smithi023 conmon[33490]: 08:40:40.829207+0000 mgr.smithi023.ehlpcl (mgr.14164) 200 : cluster [DBG] pgmap v161: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2021-11-19T08:40:42.554 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:42 smithi023 conmon[33490]: audit 2021-11-19T08:40:41.508960+0000 mon.smithi023 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.23:0/167695236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-19T08:40:42.554 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:42 smithi023 conmon[33490]: audit 2021-11-19T08:40:41.819530+0000 mon.smithi023 (mon.0) 2021-11-19T08:40:42.554 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:42 smithi023 conmon[33490]: 539 : audit [DBG] from='client.? 172.21.15.23:0/2088051968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-19T08:40:42.555 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:42 smithi023 conmon[33490]: audit 2021-11-19T08:40:42.555 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:42 smithi023 conmon[33490]: 2021-11-19 2021-11-19T08:40:42.555 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:42 smithi023 conmon[33490]: T08:40:42.165586 2021-11-19T08:40:42.555 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:42 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 2021-11-19T08:40:42.555 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:42 smithi023 conmon[33490]: 540 : audit [DBG] from='client.? 172.21.15.23:0/738411247' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:40:42.556 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:42 smithi134 conmon[36363]: cluster 2021-11-19T08:40:40.829207+0000 mgr.smithi023.ehlpcl (mgr.14164) 200 : cluster [DBG] pgmap v161: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2021-11-19T08:40:42.557 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:42 smithi134 conmon[36363]: audit 2021-11-19T08:40:41.508960+0000 mon.smithi023 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.23:0/167695236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-19T08:40:42.557 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:42 smithi134 conmon[36363]: audit 2021-11-19T08:40:41.819530+0000 mon.smithi023 (mon.0) 539 : audit 2021-11-19T08:40:42.557 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:42 smithi134 conmon[36363]: [DBG] from='client.? 172.21.15.23:0/2088051968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-19T08:40:42.557 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:42 smithi134 conmon[36363]: audit 2021-11-19T08:40:42.165586+0000 mon.smithi023 (mon.0) 540 : audit 2021-11-19T08:40:42.558 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:42 smithi134 conmon[36363]: [DBG] from='client.? 172.21.15.23:0/738411247' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:40:42.558 INFO:tasks.cephfs.mount:Ready to start KernelMount... 2021-11-19T08:40:42.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:42.559 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p /home/ubuntu/cephtest/mnt.0 2021-11-19T08:40:42.575 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-11-19T08:40:42.576 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-19T08:40:42.673 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-19T08:40:42.673 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2021-11-19T08:40:42.781 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes 2021-11-19T08:40:42.781 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,name=0,norbytes" will pass to kernel. 2021-11-19T08:40:42.785 INFO:tasks.cephfs.kernel_mount:mount command passed 2021-11-19T08:40:42.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:42.785 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2021-11-19T08:40:42.820 INFO:tasks.kclient:client.1 config is {} 2021-11-19T08:40:42.821 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2021-11-19T08:40:42.821 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi134.front.sepia.ceph.com 2021-11-19T08:40:42.821 INFO:tasks.cephfs.mount:self.client.name = client.1 2021-11-19T08:40:42.822 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2021-11-19T08:40:42.822 INFO:tasks.cephfs.mount:self.cephfs_name = None 2021-11-19T08:40:42.822 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2021-11-19T08:40:42.823 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2021-11-19T08:40:42.823 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2021-11-19T08:40:42.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:42.824 DEBUG:teuthology.orchestra.run.smithi134:> ip addr 2021-11-19T08:40:42.843 INFO:teuthology.orchestra.run.smithi134.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2021-11-19T08:40:42.843 INFO:teuthology.orchestra.run.smithi134.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2021-11-19T08:40:42.844 INFO:teuthology.orchestra.run.smithi134.stdout: inet 127.0.0.1/8 scope host lo 2021-11-19T08:40:42.844 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2021-11-19T08:40:42.844 INFO:teuthology.orchestra.run.smithi134.stdout: inet6 ::1/128 scope host 2021-11-19T08:40:42.844 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2021-11-19T08:40:42.845 INFO:teuthology.orchestra.run.smithi134.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-19T08:40:42.845 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:d9:38:e6 brd ff:ff:ff:ff:ff:ff 2021-11-19T08:40:42.845 INFO:teuthology.orchestra.run.smithi134.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-19T08:40:42.845 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:d9:38:e7 brd ff:ff:ff:ff:ff:ff 2021-11-19T08:40:42.846 INFO:teuthology.orchestra.run.smithi134.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2021-11-19T08:40:42.846 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:88:6d:e8 brd ff:ff:ff:ff:ff:ff 2021-11-19T08:40:42.847 INFO:teuthology.orchestra.run.smithi134.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2021-11-19T08:40:42.847 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:88:6d:e9 brd ff:ff:ff:ff:ff:ff 2021-11-19T08:40:42.847 INFO:teuthology.orchestra.run.smithi134.stdout: inet 172.21.15.134/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2021-11-19T08:40:42.848 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft 42226sec preferred_lft 42226sec 2021-11-19T08:40:42.848 INFO:teuthology.orchestra.run.smithi134.stdout: inet6 fe80::ec4:7aff:fe88:6de9/64 scope link 2021-11-19T08:40:42.848 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2021-11-19T08:40:42.848 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2021-11-19T08:40:42.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:42.849 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec bash -c ' 2021-11-19T08:40:42.850 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2021-11-19T08:40:42.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link add name ceph-brx type bridge 2021-11-19T08:40:42.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip addr flush dev ceph-brx 2021-11-19T08:40:42.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set ceph-brx up 2021-11-19T08:40:42.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2021-11-19T08:40:42.851 DEBUG:teuthology.orchestra.run.smithi134:> ') 2021-11-19T08:40:43.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:43.018 DEBUG:teuthology.orchestra.run.smithi134:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2021-11-19T08:40:43.045 INFO:teuthology.orchestra.run.smithi134.stdout:1 2021-11-19T08:40:43.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:43.047 DEBUG:teuthology.orchestra.run.smithi134:> route 2021-11-19T08:40:43.123 INFO:teuthology.orchestra.run.smithi134.stdout:Kernel IP routing table 2021-11-19T08:40:43.123 INFO:teuthology.orchestra.run.smithi134.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2021-11-19T08:40:43.123 INFO:teuthology.orchestra.run.smithi134.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2021-11-19T08:40:43.124 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2021-11-19T08:40:43.125 INFO:teuthology.orchestra.run.smithi134.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2021-11-19T08:40:43.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:43.125 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec bash -c ' 2021-11-19T08:40:43.126 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2021-11-19T08:40:43.126 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2021-11-19T08:40:43.126 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2021-11-19T08:40:43.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2021-11-19T08:40:43.127 DEBUG:teuthology.orchestra.run.smithi134:> ') 2021-11-19T08:40:43.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:43.411 DEBUG:teuthology.orchestra.run.smithi134:> ip netns list 2021-11-19T08:40:43.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:43.427 DEBUG:teuthology.orchestra.run.smithi134:> ip netns list-id 2021-11-19T08:40:43.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:43.482 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec bash -c ' 2021-11-19T08:40:43.483 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2021-11-19T08:40:43.483 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2021-11-19T08:40:43.483 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2021-11-19T08:40:43.484 DEBUG:teuthology.orchestra.run.smithi134:> ') 2021-11-19T08:40:43.573 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2021-11-19T08:40:43.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:43.574 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec bash -c ' 2021-11-19T08:40:43.575 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2021-11-19T08:40:43.575 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2021-11-19T08:40:43.576 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2021-11-19T08:40:43.576 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2021-11-19T08:40:43.576 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2021-11-19T08:40:43.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2021-11-19T08:40:43.577 DEBUG:teuthology.orchestra.run.smithi134:> ') 2021-11-19T08:40:43.755 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:43 smithi134 conmon[36363]: audit 2021-11-19T08:40:42.496927+0000 mon.smithi023 (mon.0 2021-11-19T08:40:43.755 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:43 smithi134 conmon[36363]: ) 541 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:43.755 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:43 smithi134 conmon[36363]: audit 2021-11-19T08:40:42.501350+0000 mon.smithi023 (mon.0) 542 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:43.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-11-19T08:40:43.785 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec bash -c ' 2021-11-19T08:40:43.785 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2021-11-19T08:40:43.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set brx.0 up 2021-11-19T08:40:43.786 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set dev brx.0 master ceph-brx 2021-11-19T08:40:43.786 DEBUG:teuthology.orchestra.run.smithi134:> ') 2021-11-19T08:40:43.889 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-11-19T08:40:43.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:43 smithi023 conmon[33490]: audit 2021-11-19T08:40:42.496927+0000 mon.smithi023 (mon.0) 541 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:43.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:43 smithi023 conmon[33490]: audit 2021-11-19 2021-11-19T08:40:43.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:43 smithi023 conmon[33490]: T08:40:42.501350+0000 mon.smithi023 (mon.0) 542 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:40:44.024 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.037+0000 7fac27b83700 1 -- 172.21.15.23:0/40461644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac200f66a0 msgr2=0x7fac200f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:44.025 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.037+0000 7fac27b83700 1 --2- 172.21.15.23:0/40461644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac200f66a0 0x7fac200f6ac0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fac10004660 tx=0x7fac10009b30).stop 2021-11-19T08:40:44.025 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.037+0000 7fac27b83700 1 -- 172.21.15.23:0/40461644 shutdown_connections 2021-11-19T08:40:44.025 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.037+0000 7fac27b83700 1 --2- 172.21.15.23:0/40461644 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac200f7940 0x7fac200f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.037+0000 7fac27b83700 1 --2- 172.21.15.23:0/40461644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac200f66a0 0x7fac200f6ac0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.037+0000 7fac27b83700 1 -- 172.21.15.23:0/40461644 >> 172.21.15.23:0/40461644 conn(0x7fac200f1da0 msgr2=0x7fac200f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.037+0000 7fac27b83700 1 -- 172.21.15.23:0/40461644 shutdown_connections 2021-11-19T08:40:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.037+0000 7fac27b83700 1 -- 172.21.15.23:0/40461644 wait complete. 2021-11-19T08:40:44.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.038+0000 7fac27b83700 1 Processor -- start 2021-11-19T08:40:44.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.038+0000 7fac27b83700 1 -- start start 2021-11-19T08:40:44.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.038+0000 7fac27b83700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac200f66a0 0x7fac20106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:44.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.038+0000 7fac27b83700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac200f7940 0x7fac20107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:44.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.038+0000 7fac27b83700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac20107940 con 0x7fac200f66a0 2021-11-19T08:40:44.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.038+0000 7fac27b83700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac20107a80 con 0x7fac200f7940 2021-11-19T08:40:44.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.038+0000 7fac2591f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac200f66a0 0x7fac20106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:44.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.038+0000 7fac2591f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac200f66a0 0x7fac20106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47160/0 (socket says 172.21.15.23:47160) 2021-11-19T08:40:44.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.038+0000 7fac2591f700 1 -- 172.21.15.23:0/1636956860 learned_addr learned my addr 172.21.15.23:0/1636956860 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:44.029 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.038+0000 7fac2511e700 1 --2- 172.21.15.23:0/1636956860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac200f7940 0x7fac20107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:44.029 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.038+0000 7fac2591f700 1 -- 172.21.15.23:0/1636956860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac200f7940 msgr2=0x7fac20107390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:44.029 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.038+0000 7fac2591f700 1 --2- 172.21.15.23:0/1636956860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac200f7940 0x7fac20107390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.029 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.038+0000 7fac2591f700 1 -- 172.21.15.23:0/1636956860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac10005040 con 0x7fac200f66a0 2021-11-19T08:40:44.030 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.039+0000 7fac2591f700 1 --2- 172.21.15.23:0/1636956860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac200f66a0 0x7fac20106ea0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fac10004e30 tx=0x7fac10009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:44.030 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.039+0000 7fac167fc700 1 -- 172.21.15.23:0/1636956860 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac10018070 con 0x7fac200f66a0 2021-11-19T08:40:44.030 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.039+0000 7fac167fc700 1 -- 172.21.15.23:0/1636956860 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac10013410 con 0x7fac200f66a0 2021-11-19T08:40:44.031 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.039+0000 7fac27b83700 1 -- 172.21.15.23:0/1636956860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2010c4c0 con 0x7fac200f66a0 2021-11-19T08:40:44.031 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.039+0000 7fac167fc700 1 -- 172.21.15.23:0/1636956860 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac1001de20 con 0x7fac200f66a0 2021-11-19T08:40:44.031 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.039+0000 7fac27b83700 1 -- 172.21.15.23:0/1636956860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2010ca80 con 0x7fac200f66a0 2021-11-19T08:40:44.031 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.040+0000 7fac27b83700 1 -- 172.21.15.23:0/1636956860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac20027a00 con 0x7fac200f66a0 2021-11-19T08:40:44.032 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.040+0000 7fac167fc700 1 -- 172.21.15.23:0/1636956860 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fac1001d420 con 0x7fac200f66a0 2021-11-19T08:40:44.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.041+0000 7fac167fc700 1 --2- 172.21.15.23:0/1636956860 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fac0c05c2a0 0x7fac0c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:44.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.041+0000 7fac167fc700 1 -- 172.21.15.23:0/1636956860 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fac10077b40 con 0x7fac200f66a0 2021-11-19T08:40:44.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.041+0000 7fac2511e700 1 --2- 172.21.15.23:0/1636956860 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fac0c05c2a0 0x7fac0c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:44.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.041+0000 7fac2511e700 1 --2- 172.21.15.23:0/1636956860 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fac0c05c2a0 0x7fac0c05e760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fac1c004c40 tx=0x7fac1c00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:44.034 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.045+0000 7fac167fc700 1 -- 172.21.15.23:0/1636956860 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac1004a800 con 0x7fac200f66a0 2021-11-19T08:40:44.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.202+0000 7fac27b83700 1 -- 172.21.15.23:0/1636956860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac200fee30 con 0x7fac200f66a0 2021-11-19T08:40:44.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.203+0000 7fac167fc700 1 -- 172.21.15.23:0/1636956860 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+14093 (secure 0 0 0) 0x7fac1004de20 con 0x7fac200f66a0 2021-11-19T08:40:44.188 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:44.188 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":41,"fsid":"908b41ee-4913-11ec-8c2c-001a4aab830c","created":"2021-11-19T08:35:14.952900+0000","modified":"2021-11-19T08:40:23.207735+0000","last_up_change":"2021-11-19T08:39:20.513380+0000","last_in_change":"2021-11-19T08:39:05.162873+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-11-19T08:38:42.679405+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2021-11-19T08:40:19.691661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2021-11-19T08:40:20.621494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"55d41e72-4389-4078-ab49-915578ae7162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6803","nonce":2382765305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6805","nonce":2382765305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6809","nonce":2382765305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2382765305},{"type":"v1","addr":"172.21.15.23:6807","nonce":2382765305}]},"public_addr":"172.21.15.23:6803/2382765305","cluster_addr":"172.21.15.23:6805/2382765305","heartbeat_back_addr":"172.21.15.23:6809/2382765305","heartbeat_front_addr":"172.21.15.23:6807/2382765305","state":["exists","up"]},{"osd":1,"uuid":"5b7aea87-24b7-43e6-9ade-8fef0e9e6dd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6811","nonce":1051412153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6813","nonce":1051412153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6817","nonce":1051412153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1051412153},{"type":"v1","addr":"172.21.15.23:6815","nonce":1051412153}]},"public_addr":"172.21.15.23:6811/1051412153","cluster_addr":"172.21.15.23:6813/1051412153","heartbeat_back_addr":"172.21.15.23:6817/1051412153","heartbeat_front_addr":"172.21.15.23:6815/1051412153","state":["exists","up"]},{"osd":2,"uuid":"e21cf6ec-4aa2-43cc-934a-bbd8d6e6694c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6819","nonce":954281054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6821","nonce":954281054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6825","nonce":954281054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":954281054},{"type":"v1","addr":"172.21.15.23:6823","nonce":954281054}]},"public_addr":"172.21.15.23:6819/954281054","cluster_addr":"172.21.15.23:6821/954281054","heartbeat_back_addr":"172.21.15.23:6825/954281054","heartbeat_front_addr":"172.21.15.23:6823/954281054","state":["exists","up"]},{"osd":3,"uuid":"276cb410-5d23-4cdf-a090-e777ebe4c82f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6801","nonce":2145743581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6803","nonce":2145743581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6807","nonce":2145743581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2145743581},{"type":"v1","addr":"172.21.15.134:6805","nonce":2145743581}]},"public_addr":"172.21.15.134:6801/2145743581","cluster_addr":"172.21.15.134:6803/2145743581","heartbeat_back_addr":"172.21.15.134:6807/2145743581","heartbeat_front_addr":"172.21.15.134:6805/2145743581","state":["exists","up"]},{"osd":4,"uuid":"015b6da1-35ed-43f4-84c9-337d2b0e79bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6809","nonce":1867765886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6811","nonce":1867765886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6815","nonce":1867765886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1867765886},{"type":"v1","addr":"172.21.15.134:6813","nonce":1867765886}]},"public_addr":"172.21.15.134:6809/1867765886","cluster_addr":"172.21.15.134:6811/1867765886","heartbeat_back_addr":"172.21.15.134:6815/1867765886","heartbeat_front_addr":"172.21.15.134:6813/1867765886","state":["exists","up"]},{"osd":5,"uuid":"86ec41d3-38df-4ee0-a320-402055f7895b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6817","nonce":3784199400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6819","nonce":3784199400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6823","nonce":3784199400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3784199400},{"type":"v1","addr":"172.21.15.134:6821","nonce":3784199400}]},"public_addr":"172.21.15.134:6817/3784199400","cluster_addr":"172.21.15.134:6819/3784199400","heartbeat_back_addr":"172.21.15.134:6823/3784199400","heartbeat_front_addr":"172.21.15.134:6821/3784199400","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:37:52.252795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:12.408238+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:31.280017+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:38:47.181003+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:39:03.121900+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-11-19T08:39:19.383582+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/192042719":"2021-11-20T08:36:18.730761+0000","172.21.15.23:0/4146945986":"2021-11-20T08:36:18.730761+0000","172.21.15.23:6801/158654977":"2021-11-20T08:36:18.730761+0000","172.21.15.23:6800/158654977":"2021-11-20T08:36:18.730761+0000","172.21.15.23:0/992169182":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/1001097146":"2021-11-20T08:36:02.243722+0000","172.21.15.23:6800/1352671878":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/3039325239":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6801/1352671878":"2021-11-20T08:36:02.243722+0000","172.21.15.23:0/4131282839":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6800/945698991":"2021-11-20T08:35:42.544921+0000","172.21.15.23:6801/945698991":"2021-11-20T08:35:42.544921+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-11-19T08:40:44.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.205+0000 7fac27b83700 1 -- 172.21.15.23:0/1636956860 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fac0c05c2a0 msgr2=0x7fac0c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:44.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.205+0000 7fac27b83700 1 --2- 172.21.15.23:0/1636956860 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fac0c05c2a0 0x7fac0c05e760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fac1c004c40 tx=0x7fac1c00b040).stop 2021-11-19T08:40:44.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.205+0000 7fac27b83700 1 -- 172.21.15.23:0/1636956860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac200f66a0 msgr2=0x7fac20106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:44.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.205+0000 7fac27b83700 1 --2- 172.21.15.23:0/1636956860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac200f66a0 0x7fac20106ea0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fac10004e30 tx=0x7fac10009b30).stop 2021-11-19T08:40:44.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.205+0000 7fac27b83700 1 -- 172.21.15.23:0/1636956860 shutdown_connections 2021-11-19T08:40:44.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.205+0000 7fac27b83700 1 --2- 172.21.15.23:0/1636956860 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fac0c05c2a0 0x7fac0c05e760 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.205+0000 7fac27b83700 1 --2- 172.21.15.23:0/1636956860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac200f66a0 0x7fac20106ea0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.205+0000 7fac27b83700 1 --2- 172.21.15.23:0/1636956860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac200f7940 0x7fac20107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.192 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.205+0000 7fac27b83700 1 -- 172.21.15.23:0/1636956860 >> 172.21.15.23:0/1636956860 conn(0x7fac200f1da0 msgr2=0x7fac200fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:44.192 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.205+0000 7fac27b83700 1 -- 172.21.15.23:0/1636956860 shutdown_connections 2021-11-19T08:40:44.192 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.205+0000 7fac27b83700 1 -- 172.21.15.23:0/1636956860 wait complete. 2021-11-19T08:40:44.202 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-11-19T08:38:42.679405+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2021-11-19T08:40:19.691661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2021-11-19T08:40:20.621494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-11-19T08:40:44.203 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2021-11-19T08:40:44.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.350+0000 7faae23d7700 1 -- 172.21.15.23:0/2930550494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc0f5580 msgr2=0x7faadc0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:44.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.350+0000 7faae23d7700 1 --2- 172.21.15.23:0/2930550494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc0f5580 0x7faadc0f59a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7faacc004660 tx=0x7faacc009b30).stop 2021-11-19T08:40:44.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.351+0000 7faae23d7700 1 -- 172.21.15.23:0/2930550494 shutdown_connections 2021-11-19T08:40:44.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.351+0000 7faae23d7700 1 --2- 172.21.15.23:0/2930550494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faadc0f6820 0x7faadc0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.351+0000 7faae23d7700 1 --2- 172.21.15.23:0/2930550494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc0f5580 0x7faadc0f59a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.351+0000 7faae23d7700 1 -- 172.21.15.23:0/2930550494 >> 172.21.15.23:0/2930550494 conn(0x7faadc0f0ce0 msgr2=0x7faadc0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:44.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.351+0000 7faae23d7700 1 -- 172.21.15.23:0/2930550494 shutdown_connections 2021-11-19T08:40:44.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.351+0000 7faae23d7700 1 -- 172.21.15.23:0/2930550494 wait complete. 2021-11-19T08:40:44.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.351+0000 7faae23d7700 1 Processor -- start 2021-11-19T08:40:44.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faae23d7700 1 -- start start 2021-11-19T08:40:44.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faae23d7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc0f5580 0x7faadc105d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:44.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faae23d7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faadc0f6820 0x7faadc106220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:44.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faae23d7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc1067d0 con 0x7faadc0f5580 2021-11-19T08:40:44.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faae23d7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc106910 con 0x7faadc0f6820 2021-11-19T08:40:44.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faadbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc0f5580 0x7faadc105d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:44.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faadbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc0f5580 0x7faadc105d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47170/0 (socket says 172.21.15.23:47170) 2021-11-19T08:40:44.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faadbfff700 1 -- 172.21.15.23:0/296349821 learned_addr learned my addr 172.21.15.23:0/296349821 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:44.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faadb7fe700 1 --2- 172.21.15.23:0/296349821 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faadc0f6820 0x7faadc106220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:44.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faadbfff700 1 -- 172.21.15.23:0/296349821 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faadc0f6820 msgr2=0x7faadc106220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:44.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faadbfff700 1 --2- 172.21.15.23:0/296349821 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faadc0f6820 0x7faadc106220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faadbfff700 1 -- 172.21.15.23:0/296349821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faacc005040 con 0x7faadc0f5580 2021-11-19T08:40:44.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faadbfff700 1 --2- 172.21.15.23:0/296349821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc0f5580 0x7faadc105d30 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7faacc000c00 tx=0x7faacc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:44.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faad8ff9700 1 -- 172.21.15.23:0/296349821 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faacc019070 con 0x7faadc0f5580 2021-11-19T08:40:44.344 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.352+0000 7faae23d7700 1 -- 172.21.15.23:0/296349821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faadc10b350 con 0x7faadc0f5580 2021-11-19T08:40:44.344 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.353+0000 7faad8ff9700 1 -- 172.21.15.23:0/296349821 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faacc007e70 con 0x7faadc0f5580 2021-11-19T08:40:44.344 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.353+0000 7faae23d7700 1 -- 172.21.15.23:0/296349821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc10b910 con 0x7faadc0f5580 2021-11-19T08:40:44.345 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.353+0000 7faad8ff9700 1 -- 172.21.15.23:0/296349821 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faacc014d10 con 0x7faadc0f5580 2021-11-19T08:40:44.345 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.353+0000 7faae23d7700 1 -- 172.21.15.23:0/296349821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc046330 con 0x7faadc0f5580 2021-11-19T08:40:44.345 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.354+0000 7faad8ff9700 1 -- 172.21.15.23:0/296349821 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7faacc025070 con 0x7faadc0f5580 2021-11-19T08:40:44.345 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.354+0000 7faad8ff9700 1 --2- 172.21.15.23:0/296349821 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7faac405c2a0 0x7faac405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:44.346 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.354+0000 7faad8ff9700 1 -- 172.21.15.23:0/296349821 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7faacc076d90 con 0x7faadc0f5580 2021-11-19T08:40:44.346 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.354+0000 7faadb7fe700 1 --2- 172.21.15.23:0/296349821 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7faac405c2a0 0x7faac405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:44.346 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.355+0000 7faadb7fe700 1 --2- 172.21.15.23:0/296349821 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7faac405c2a0 0x7faac405e760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7faad0009fd0 tx=0x7faad000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:44.347 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.358+0000 7faad8ff9700 1 -- 172.21.15.23:0/296349821 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faacc049a50 con 0x7faadc0f5580 2021-11-19T08:40:44.504 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.518+0000 7faae23d7700 1 -- 172.21.15.23:0/296349821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7faadc027a00 con 0x7faadc0f5580 2021-11-19T08:40:44.504 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.519+0000 7faad8ff9700 1 -- 172.21.15.23:0/296349821 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v41) v1 ==== 110+0+10 (secure 0 0 0) 0x7faacc04d070 con 0x7faadc0f5580 2021-11-19T08:40:44.505 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2021-11-19T08:40:44.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.521+0000 7faae23d7700 1 -- 172.21.15.23:0/296349821 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7faac405c2a0 msgr2=0x7faac405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:44.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.521+0000 7faae23d7700 1 --2- 172.21.15.23:0/296349821 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7faac405c2a0 0x7faac405e760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7faad0009fd0 tx=0x7faad000b040).stop 2021-11-19T08:40:44.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.521+0000 7faae23d7700 1 -- 172.21.15.23:0/296349821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc0f5580 msgr2=0x7faadc105d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:44.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.521+0000 7faae23d7700 1 --2- 172.21.15.23:0/296349821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc0f5580 0x7faadc105d30 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7faacc000c00 tx=0x7faacc00b040).stop 2021-11-19T08:40:44.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.521+0000 7faae23d7700 1 -- 172.21.15.23:0/296349821 shutdown_connections 2021-11-19T08:40:44.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.521+0000 7faae23d7700 1 --2- 172.21.15.23:0/296349821 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7faac405c2a0 0x7faac405e760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.521+0000 7faae23d7700 1 --2- 172.21.15.23:0/296349821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc0f5580 0x7faadc105d30 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.521+0000 7faae23d7700 1 --2- 172.21.15.23:0/296349821 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faadc0f6820 0x7faadc106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.521+0000 7faae23d7700 1 -- 172.21.15.23:0/296349821 >> 172.21.15.23:0/296349821 conn(0x7faadc0f0ce0 msgr2=0x7faadc0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:44.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.521+0000 7faae23d7700 1 -- 172.21.15.23:0/296349821 shutdown_connections 2021-11-19T08:40:44.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.521+0000 7faae23d7700 1 -- 172.21.15.23:0/296349821 wait complete. 2021-11-19T08:40:44.518 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2021-11-19T08:40:44.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.665+0000 7f821ef81700 1 -- 172.21.15.23:0/1260283405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82180f6b80 msgr2=0x7f82180f6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:44.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.665+0000 7f821ef81700 1 --2- 172.21.15.23:0/1260283405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82180f6b80 0x7f82180f6fa0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f8210004660 tx=0x7f8210009b30).stop 2021-11-19T08:40:44.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.665+0000 7f821ef81700 1 -- 172.21.15.23:0/1260283405 shutdown_connections 2021-11-19T08:40:44.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.665+0000 7f821ef81700 1 --2- 172.21.15.23:0/1260283405 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82180f5880 0x7f82180f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.665+0000 7f821ef81700 1 --2- 172.21.15.23:0/1260283405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82180f6b80 0x7f82180f6fa0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.654 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.665+0000 7f821ef81700 1 -- 172.21.15.23:0/1260283405 >> 172.21.15.23:0/1260283405 conn(0x7f82180f0470 msgr2=0x7f82180f28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:44.654 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.665+0000 7f821ef81700 1 -- 172.21.15.23:0/1260283405 shutdown_connections 2021-11-19T08:40:44.654 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.665+0000 7f821ef81700 1 -- 172.21.15.23:0/1260283405 wait complete. 2021-11-19T08:40:44.654 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.666+0000 7f821ef81700 1 Processor -- start 2021-11-19T08:40:44.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.666+0000 7f821ef81700 1 -- start start 2021-11-19T08:40:44.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.666+0000 7f821ef81700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82180f5880 0x7f82180f5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:44.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.666+0000 7f821ef81700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82180f6b80 0x7f821810bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:44.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.666+0000 7f821ef81700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821810c1e0 con 0x7f82180f5880 2021-11-19T08:40:44.656 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.666+0000 7f821ef81700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821810c320 con 0x7f82180f6b80 2021-11-19T08:40:44.656 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.666+0000 7f821cd1d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82180f5880 0x7f82180f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:44.656 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.666+0000 7f821cd1d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82180f5880 0x7f82180f5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47184/0 (socket says 172.21.15.23:47184) 2021-11-19T08:40:44.656 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.666+0000 7f821cd1d700 1 -- 172.21.15.23:0/1532453009 learned_addr learned my addr 172.21.15.23:0/1532453009 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:44.656 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.667+0000 7f8217fff700 1 --2- 172.21.15.23:0/1532453009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82180f6b80 0x7f821810bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:44.657 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.667+0000 7f821cd1d700 1 -- 172.21.15.23:0/1532453009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82180f6b80 msgr2=0x7f821810bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:44.657 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.667+0000 7f821cd1d700 1 --2- 172.21.15.23:0/1532453009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82180f6b80 0x7f821810bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.657 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.667+0000 7f821cd1d700 1 -- 172.21.15.23:0/1532453009 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8210005040 con 0x7f82180f5880 2021-11-19T08:40:44.657 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.667+0000 7f8217fff700 1 --2- 172.21.15.23:0/1532453009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82180f6b80 0x7f821810bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:40:44.658 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.667+0000 7f821cd1d700 1 --2- 172.21.15.23:0/1532453009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82180f5880 0x7f82180f5220 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f8210004e30 tx=0x7f8210009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:44.658 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.667+0000 7f82157fa700 1 -- 172.21.15.23:0/1532453009 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8210018070 con 0x7f82180f5880 2021-11-19T08:40:44.658 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.667+0000 7f821ef81700 1 -- 172.21.15.23:0/1532453009 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821810c520 con 0x7f82180f5880 2021-11-19T08:40:44.658 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.667+0000 7f82157fa700 1 -- 172.21.15.23:0/1532453009 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8210013410 con 0x7f82180f5880 2021-11-19T08:40:44.659 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.667+0000 7f82157fa700 1 -- 172.21.15.23:0/1532453009 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f821001c3c0 con 0x7f82180f5880 2021-11-19T08:40:44.659 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.667+0000 7f821ef81700 1 -- 172.21.15.23:0/1532453009 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821810ca50 con 0x7f82180f5880 2021-11-19T08:40:44.659 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.668+0000 7f821ef81700 1 -- 172.21.15.23:0/1532453009 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8218027a00 con 0x7f82180f5880 2021-11-19T08:40:44.660 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.668+0000 7f82157fa700 1 -- 172.21.15.23:0/1532453009 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f821001d420 con 0x7f82180f5880 2021-11-19T08:40:44.660 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.669+0000 7f82157fa700 1 --2- 172.21.15.23:0/1532453009 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f820405c2a0 0x7f820405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:44.661 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.669+0000 7f82157fa700 1 -- 172.21.15.23:0/1532453009 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f8210077ac0 con 0x7f82180f5880 2021-11-19T08:40:44.661 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.669+0000 7f8217fff700 1 --2- 172.21.15.23:0/1532453009 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f820405c2a0 0x7f820405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:44.661 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.669+0000 7f8217fff700 1 --2- 172.21.15.23:0/1532453009 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f820405c2a0 0x7f820405e760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f82180f6950 tx=0x7f820800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:44.661 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.673+0000 7f82157fa700 1 -- 172.21.15.23:0/1532453009 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f821004a780 con 0x7f82180f5880 2021-11-19T08:40:44.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:44 smithi134 conmon[36363]: cluster 2021-11-19T08:40:42.829634+0000 mgr.smithi023.ehlpcl (mgr.14164) 201 : cluster [DBG] pgmap v162: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 7 op/s 2021-11-19T08:40:44.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:44 smithi134 conmon[36363]: audit 2021-11-19T08:40:44.203512+0000 mon.smithi023 (mon.0) 543 : audit [DBG] from='client.? 172.21.15.23:0/1636956860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-19T08:40:44.816 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.831+0000 7f821ef81700 1 -- 172.21.15.23:0/1532453009 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f82180fdd70 con 0x7f82180f5880 2021-11-19T08:40:44.816 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.831+0000 7f82157fa700 1 -- 172.21.15.23:0/1532453009 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7f821004dda0 con 0x7f82180f5880 2021-11-19T08:40:44.816 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2021-11-19T08:40:44.818 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.833+0000 7f821ef81700 1 -- 172.21.15.23:0/1532453009 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f820405c2a0 msgr2=0x7f820405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:44.818 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.833+0000 7f821ef81700 1 --2- 172.21.15.23:0/1532453009 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f820405c2a0 0x7f820405e760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f82180f6950 tx=0x7f820800b040).stop 2021-11-19T08:40:44.818 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.833+0000 7f821ef81700 1 -- 172.21.15.23:0/1532453009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82180f5880 msgr2=0x7f82180f5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:44.819 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.833+0000 7f821ef81700 1 --2- 172.21.15.23:0/1532453009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82180f5880 0x7f82180f5220 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f8210004e30 tx=0x7f8210009b30).stop 2021-11-19T08:40:44.819 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.833+0000 7f821ef81700 1 -- 172.21.15.23:0/1532453009 shutdown_connections 2021-11-19T08:40:44.819 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.833+0000 7f821ef81700 1 --2- 172.21.15.23:0/1532453009 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f820405c2a0 0x7f820405e760 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.819 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.833+0000 7f821ef81700 1 --2- 172.21.15.23:0/1532453009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82180f5880 0x7f82180f5220 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.819 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.833+0000 7f821ef81700 1 --2- 172.21.15.23:0/1532453009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82180f6b80 0x7f821810bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.820 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.833+0000 7f821ef81700 1 -- 172.21.15.23:0/1532453009 >> 172.21.15.23:0/1532453009 conn(0x7f82180f0470 msgr2=0x7f82180fa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:44.820 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.833+0000 7f821ef81700 1 -- 172.21.15.23:0/1532453009 shutdown_connections 2021-11-19T08:40:44.820 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.833+0000 7f821ef81700 1 -- 172.21.15.23:0/1532453009 wait complete. 2021-11-19T08:40:44.831 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:44 smithi023 conmon[33490]: cluster 2021-11-19T08:40:42.829634 2021-11-19T08:40:44.831 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:44 smithi023 conmon[33490]: +0000 mgr.smithi023.ehlpcl (mgr.14164) 201 : cluster [DBG] pgmap v162: 65 pgs: 65 active+clean; 3.6 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 7 op/s 2021-11-19T08:40:44.832 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:44 smithi023 conmon[33490]: audit 2021-11-19T08:40:44.203512+0000 mon.smithi023 (mon.0) 543 : audit [DBG] from='client.? 172.21.15.23:0/1636956860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-11-19T08:40:44.833 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2021-11-19T08:40:44.960 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.971+0000 7f88828c4700 1 -- 172.21.15.23:0/3028258276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f887c0f4360 msgr2=0x7f887c0f4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:44.960 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.971+0000 7f88828c4700 1 --2- 172.21.15.23:0/3028258276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f887c0f4360 0x7f887c0f4780 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f886c004690 tx=0x7f886c009b30).stop 2021-11-19T08:40:44.960 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.971+0000 7f88828c4700 1 -- 172.21.15.23:0/3028258276 shutdown_connections 2021-11-19T08:40:44.961 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.971+0000 7f88828c4700 1 --2- 172.21.15.23:0/3028258276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f887c0f4c70 0x7f887c0fbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.961 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.971+0000 7f88828c4700 1 --2- 172.21.15.23:0/3028258276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f887c0f4360 0x7f887c0f4780 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.961 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.971+0000 7f88828c4700 1 -- 172.21.15.23:0/3028258276 >> 172.21.15.23:0/3028258276 conn(0x7f887c0f0020 msgr2=0x7f887c0f2440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:44.961 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.971+0000 7f88828c4700 1 -- 172.21.15.23:0/3028258276 shutdown_connections 2021-11-19T08:40:44.962 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.971+0000 7f88828c4700 1 -- 172.21.15.23:0/3028258276 wait complete. 2021-11-19T08:40:44.962 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.971+0000 7f88828c4700 1 Processor -- start 2021-11-19T08:40:44.962 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.971+0000 7f88828c4700 1 -- start start 2021-11-19T08:40:44.962 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.971+0000 7f88828c4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f887c0f4360 0x7f887c106dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:44.963 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.971+0000 7f88828c4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f887c0f4c70 0x7f887c1072b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:44.963 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.971+0000 7f88828c4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c107860 con 0x7f887c0f4360 2021-11-19T08:40:44.963 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.971+0000 7f88828c4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c1079a0 con 0x7f887c0f4c70 2021-11-19T08:40:44.963 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.972+0000 7f887bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f887c0f4360 0x7f887c106dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:44.964 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.972+0000 7f887bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f887c0f4360 0x7f887c106dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47192/0 (socket says 172.21.15.23:47192) 2021-11-19T08:40:44.964 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.972+0000 7f887bfff700 1 -- 172.21.15.23:0/3521975289 learned_addr learned my addr 172.21.15.23:0/3521975289 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:44.964 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.972+0000 7f887b7fe700 1 --2- 172.21.15.23:0/3521975289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f887c0f4c70 0x7f887c1072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:44.964 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.972+0000 7f887bfff700 1 -- 172.21.15.23:0/3521975289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f887c0f4c70 msgr2=0x7f887c1072b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:44.965 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.972+0000 7f887bfff700 1 --2- 172.21.15.23:0/3521975289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f887c0f4c70 0x7f887c1072b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:44.965 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.972+0000 7f887bfff700 1 -- 172.21.15.23:0/3521975289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c005040 con 0x7f887c0f4360 2021-11-19T08:40:44.965 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.972+0000 7f887b7fe700 1 --2- 172.21.15.23:0/3521975289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f887c0f4c70 0x7f887c1072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:40:44.966 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.972+0000 7f887bfff700 1 --2- 172.21.15.23:0/3521975289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f887c0f4360 0x7f887c106dc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f886c007a60 tx=0x7f886c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:44.966 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.972+0000 7f8878ff9700 1 -- 172.21.15.23:0/3521975289 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f886c018070 con 0x7f887c0f4360 2021-11-19T08:40:44.966 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.972+0000 7f88828c4700 1 -- 172.21.15.23:0/3521975289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f887c10c3e0 con 0x7f887c0f4360 2021-11-19T08:40:44.966 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.972+0000 7f8878ff9700 1 -- 172.21.15.23:0/3521975289 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f886c004ed0 con 0x7f887c0f4360 2021-11-19T08:40:44.967 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.972+0000 7f8878ff9700 1 -- 172.21.15.23:0/3521975289 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f886c01c620 con 0x7f887c0f4360 2021-11-19T08:40:44.967 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.972+0000 7f88828c4700 1 -- 172.21.15.23:0/3521975289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887c10c920 con 0x7f887c0f4360 2021-11-19T08:40:44.967 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.973+0000 7f88828c4700 1 -- 172.21.15.23:0/3521975289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887c027a00 con 0x7f887c0f4360 2021-11-19T08:40:44.968 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.974+0000 7f8878ff9700 1 -- 172.21.15.23:0/3521975289 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f886c01c890 con 0x7f887c0f4360 2021-11-19T08:40:44.968 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.974+0000 7f8878ff9700 1 --2- 172.21.15.23:0/3521975289 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f886405c2a0 0x7f886405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:44.968 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.974+0000 7f8878ff9700 1 -- 172.21.15.23:0/3521975289 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f886c077c50 con 0x7f887c0f4360 2021-11-19T08:40:44.968 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.974+0000 7f887b7fe700 1 --2- 172.21.15.23:0/3521975289 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f886405c2a0 0x7f886405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:44.969 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.974+0000 7f887b7fe700 1 --2- 172.21.15.23:0/3521975289 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f886405c2a0 0x7f886405e760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f8870009760 tx=0x7f8870009c60).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:44.969 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:44.978+0000 7f8878ff9700 1 -- 172.21.15.23:0/3521975289 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f886c04a910 con 0x7f887c0f4360 2021-11-19T08:40:45.125 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.140+0000 7f88828c4700 1 -- 172.21.15.23:0/3521975289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f887c10c570 con 0x7f887c0f4360 2021-11-19T08:40:45.125 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.140+0000 7f8878ff9700 1 -- 172.21.15.23:0/3521975289 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v41) v1 ==== 107+0+11 (secure 0 0 0) 0x7f887c10c570 con 0x7f887c0f4360 2021-11-19T08:40:45.126 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2021-11-19T08:40:45.127 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.142+0000 7f88828c4700 1 -- 172.21.15.23:0/3521975289 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f886405c2a0 msgr2=0x7f886405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:45.127 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.142+0000 7f88828c4700 1 --2- 172.21.15.23:0/3521975289 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f886405c2a0 0x7f886405e760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f8870009760 tx=0x7f8870009c60).stop 2021-11-19T08:40:45.128 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.142+0000 7f88828c4700 1 -- 172.21.15.23:0/3521975289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f887c0f4360 msgr2=0x7f887c106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:45.128 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.142+0000 7f88828c4700 1 --2- 172.21.15.23:0/3521975289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f887c0f4360 0x7f887c106dc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f886c007a60 tx=0x7f886c009b30).stop 2021-11-19T08:40:45.128 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.142+0000 7f88828c4700 1 -- 172.21.15.23:0/3521975289 shutdown_connections 2021-11-19T08:40:45.128 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.142+0000 7f88828c4700 1 --2- 172.21.15.23:0/3521975289 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f886405c2a0 0x7f886405e760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.129 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.142+0000 7f88828c4700 1 --2- 172.21.15.23:0/3521975289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f887c0f4360 0x7f887c106dc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.129 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.142+0000 7f88828c4700 1 --2- 172.21.15.23:0/3521975289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f887c0f4c70 0x7f887c1072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.129 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.142+0000 7f88828c4700 1 -- 172.21.15.23:0/3521975289 >> 172.21.15.23:0/3521975289 conn(0x7f887c0f0020 msgr2=0x7f887c0f8730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:45.129 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.142+0000 7f88828c4700 1 -- 172.21.15.23:0/3521975289 shutdown_connections 2021-11-19T08:40:45.130 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.142+0000 7f88828c4700 1 -- 172.21.15.23:0/3521975289 wait complete. 2021-11-19T08:40:45.139 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-11-19T08:40:45.140 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:40:45.263 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.276+0000 7f62cdda5700 1 -- 172.21.15.23:0/674539305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62c80f54a0 msgr2=0x7f62c80f58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:45.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.276+0000 7f62cdda5700 1 --2- 172.21.15.23:0/674539305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62c80f54a0 0x7f62c80f58c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f62bc004660 tx=0x7f62bc009b30).stop 2021-11-19T08:40:45.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.276+0000 7f62cdda5700 1 -- 172.21.15.23:0/674539305 shutdown_connections 2021-11-19T08:40:45.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.276+0000 7f62cdda5700 1 --2- 172.21.15.23:0/674539305 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62c80f6740 0x7f62c80f6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.276+0000 7f62cdda5700 1 --2- 172.21.15.23:0/674539305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62c80f54a0 0x7f62c80f58c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.276+0000 7f62cdda5700 1 -- 172.21.15.23:0/674539305 >> 172.21.15.23:0/674539305 conn(0x7f62c80f0c00 msgr2=0x7f62c80f3020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:45.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.276+0000 7f62cdda5700 1 -- 172.21.15.23:0/674539305 shutdown_connections 2021-11-19T08:40:45.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.276+0000 7f62cdda5700 1 -- 172.21.15.23:0/674539305 wait complete. 2021-11-19T08:40:45.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.276+0000 7f62cdda5700 1 Processor -- start 2021-11-19T08:40:45.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.276+0000 7f62cdda5700 1 -- start start 2021-11-19T08:40:45.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62cdda5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62c80f54a0 0x7f62c8105d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:45.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62cdda5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62c80f6740 0x7f62c8106200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:45.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62cdda5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c81067b0 con 0x7f62c80f6740 2021-11-19T08:40:45.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62cdda5700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c8106fb0 con 0x7f62c80f54a0 2021-11-19T08:40:45.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62c7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62c80f6740 0x7f62c8106200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:45.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62c7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62c80f6740 0x7f62c8106200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47206/0 (socket says 172.21.15.23:47206) 2021-11-19T08:40:45.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62c7fff700 1 -- 172.21.15.23:0/2115787125 learned_addr learned my addr 172.21.15.23:0/2115787125 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:45.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62ccda3700 1 --2- 172.21.15.23:0/2115787125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62c80f54a0 0x7f62c8105d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:45.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62c7fff700 1 -- 172.21.15.23:0/2115787125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62c80f54a0 msgr2=0x7f62c8105d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:45.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62c7fff700 1 --2- 172.21.15.23:0/2115787125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62c80f54a0 0x7f62c8105d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62c7fff700 1 -- 172.21.15.23:0/2115787125 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc005040 con 0x7f62c80f6740 2021-11-19T08:40:45.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62c7fff700 1 --2- 172.21.15.23:0/2115787125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62c80f6740 0x7f62c8106200 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f62b400c530 tx=0x7f62b4007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:45.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62c57fa700 1 -- 172.21.15.23:0/2115787125 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62b400f650 con 0x7f62c80f6740 2021-11-19T08:40:45.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62c57fa700 1 -- 172.21.15.23:0/2115787125 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62b400fc60 con 0x7f62c80f6740 2021-11-19T08:40:45.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62cdda5700 1 -- 172.21.15.23:0/2115787125 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c8107290 con 0x7f62c80f6740 2021-11-19T08:40:45.270 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62c57fa700 1 -- 172.21.15.23:0/2115787125 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62b4012c80 con 0x7f62c80f6740 2021-11-19T08:40:45.270 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.277+0000 7f62cdda5700 1 -- 172.21.15.23:0/2115787125 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c8107850 con 0x7f62c80f6740 2021-11-19T08:40:45.271 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.278+0000 7f62cdda5700 1 -- 172.21.15.23:0/2115787125 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c80f4890 con 0x7f62c80f6740 2021-11-19T08:40:45.271 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.279+0000 7f62c57fa700 1 -- 172.21.15.23:0/2115787125 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f62b4010070 con 0x7f62c80f6740 2021-11-19T08:40:45.272 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.279+0000 7f62c57fa700 1 --2- 172.21.15.23:0/2115787125 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b805c300 0x7f62b805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:45.272 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.279+0000 7f62c57fa700 1 -- 172.21.15.23:0/2115787125 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f62b4075c00 con 0x7f62c80f6740 2021-11-19T08:40:45.273 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.279+0000 7f62ccda3700 1 --2- 172.21.15.23:0/2115787125 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b805c300 0x7f62b805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:45.273 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.280+0000 7f62ccda3700 1 --2- 172.21.15.23:0/2115787125 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b805c300 0x7f62b805e7c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f62bc009d60 tx=0x7f62bc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:45.273 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.283+0000 7f62c57fa700 1 -- 172.21.15.23:0/2115787125 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62b40452b0 con 0x7f62c80f6740 2021-11-19T08:40:45.468 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.483+0000 7f62cdda5700 1 -- 172.21.15.23:0/2115787125 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62c8045ad0 con 0x7f62c80f6740 2021-11-19T08:40:45.468 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.483+0000 7f62c57fa700 1 -- 172.21.15.23:0/2115787125 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2962 (secure 0 0 0) 0x7f62b407b020 con 0x7f62c80f6740 2021-11-19T08:40:45.469 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:40:45.470 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":14,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:40:37.178115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14434,"mds_1":24283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14434":{"gid":14434,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.23:6827/530484646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":530484646},{"type":"v1","addr":"172.21.15.23:6827","nonce":530484646}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14446":{"gid":14446,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.23:6829/2176063367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2176063367},{"type":"v1","addr":"172.21.15.23:6829","nonce":2176063367}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24271":{"gid":24271,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":3,"addr":"172.21.15.134:6825/1874302373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1874302373},{"type":"v1","addr":"172.21.15.134:6825","nonce":1874302373}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24283":{"gid":24283,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":8,"state":"up:active","state_seq":2,"addr":"172.21.15.134:6827/3905671987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3905671987},{"type":"v1","addr":"172.21.15.134:6827","nonce":3905671987}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:40:45.472 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.485+0000 7f62cdda5700 1 -- 172.21.15.23:0/2115787125 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b805c300 msgr2=0x7f62b805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:45.472 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.485+0000 7f62cdda5700 1 --2- 172.21.15.23:0/2115787125 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b805c300 0x7f62b805e7c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f62bc009d60 tx=0x7f62bc009b30).stop 2021-11-19T08:40:45.472 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.485+0000 7f62cdda5700 1 -- 172.21.15.23:0/2115787125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62c80f6740 msgr2=0x7f62c8106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:45.473 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.485+0000 7f62cdda5700 1 --2- 172.21.15.23:0/2115787125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62c80f6740 0x7f62c8106200 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f62b400c530 tx=0x7f62b4007e20).stop 2021-11-19T08:40:45.473 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.485+0000 7f62cdda5700 1 -- 172.21.15.23:0/2115787125 shutdown_connections 2021-11-19T08:40:45.473 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.485+0000 7f62cdda5700 1 --2- 172.21.15.23:0/2115787125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62c80f54a0 0x7f62c8105d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.473 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.485+0000 7f62cdda5700 1 --2- 172.21.15.23:0/2115787125 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f62b805c300 0x7f62b805e7c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.473 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.485+0000 7f62cdda5700 1 --2- 172.21.15.23:0/2115787125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62c80f6740 0x7f62c8106200 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.475 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.485+0000 7f62cdda5700 1 -- 172.21.15.23:0/2115787125 >> 172.21.15.23:0/2115787125 conn(0x7f62c80f0c00 msgr2=0x7f62c80fa190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:45.475 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.485+0000 7f62cdda5700 1 -- 172.21.15.23:0/2115787125 shutdown_connections 2021-11-19T08:40:45.475 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.485+0000 7f62cdda5700 1 -- 172.21.15.23:0/2115787125 wait complete. 2021-11-19T08:40:45.475 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 14 2021-11-19T08:40:45.483 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:40:37.178115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14434, 'mds_1': 24283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14434': {'gid': 14434, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.23:6827/530484646', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 530484646}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 530484646}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14446': {'gid': 14446, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 3, 'addr': '172.21.15.23:6829/2176063367', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 2176063367}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 2176063367}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24271': {'gid': 24271, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 3, 'addr': '172.21.15.134:6825/1874302373', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 1874302373}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 1874302373}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24283': {'gid': 24283, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 8, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.134:6827/3905671987', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3905671987}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3905671987}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:40:45.483 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2021-11-19T08:40:45.483 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.cephfs.smithi023.fuvzkr status 2021-11-19T08:40:45.605 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.618+0000 7fc7b32f0700 1 -- 172.21.15.23:0/3317569518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7ac0f6820 msgr2=0x7fc7ac0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:45.605 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.618+0000 7fc7b32f0700 1 --2- 172.21.15.23:0/3317569518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7ac0f6820 0x7fc7ac0f6c80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fc79c004660 tx=0x7fc79c009b30).stop 2021-11-19T08:40:45.605 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.618+0000 7fc7b32f0700 1 -- 172.21.15.23:0/3317569518 shutdown_connections 2021-11-19T08:40:45.606 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.618+0000 7fc7b32f0700 1 --2- 172.21.15.23:0/3317569518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7ac0f6820 0x7fc7ac0f6c80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.606 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.618+0000 7fc7b32f0700 1 --2- 172.21.15.23:0/3317569518 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7ac0f5580 0x7fc7ac0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.606 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.618+0000 7fc7b32f0700 1 -- 172.21.15.23:0/3317569518 >> 172.21.15.23:0/3317569518 conn(0x7fc7ac0f0ce0 msgr2=0x7fc7ac0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:45.606 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.618+0000 7fc7b32f0700 1 -- 172.21.15.23:0/3317569518 shutdown_connections 2021-11-19T08:40:45.607 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.618+0000 7fc7b32f0700 1 -- 172.21.15.23:0/3317569518 wait complete. 2021-11-19T08:40:45.607 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.618+0000 7fc7b32f0700 1 Processor -- start 2021-11-19T08:40:45.607 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.618+0000 7fc7b32f0700 1 -- start start 2021-11-19T08:40:45.607 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7b32f0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7ac0f5580 0x7fc7ac05ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:45.607 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7b32f0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7ac0f6820 0x7fc7ac10e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:45.608 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7b32f0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac107a70 con 0x7fc7ac0f5580 2021-11-19T08:40:45.608 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7b32f0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac10e580 con 0x7fc7ac0f6820 2021-11-19T08:40:45.608 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7b108c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7ac0f5580 0x7fc7ac05ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:45.608 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7b108c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7ac0f5580 0x7fc7ac05ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47214/0 (socket says 172.21.15.23:47214) 2021-11-19T08:40:45.609 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7b108c700 1 -- 172.21.15.23:0/2236819929 learned_addr learned my addr 172.21.15.23:0/2236819929 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:45.609 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7b088b700 1 --2- 172.21.15.23:0/2236819929 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7ac0f6820 0x7fc7ac10e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:45.609 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7b108c700 1 -- 172.21.15.23:0/2236819929 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7ac0f6820 msgr2=0x7fc7ac10e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:45.609 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7b108c700 1 --2- 172.21.15.23:0/2236819929 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7ac0f6820 0x7fc7ac10e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.609 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7b108c700 1 -- 172.21.15.23:0/2236819929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c005040 con 0x7fc7ac0f5580 2021-11-19T08:40:45.610 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7b108c700 1 --2- 172.21.15.23:0/2236819929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7ac0f5580 0x7fc7ac05ff00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fc7a80026d0 tx=0x7fc7a8009dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:45.610 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7a1ffb700 1 -- 172.21.15.23:0/2236819929 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7a80079c0 con 0x7fc7ac0f5580 2021-11-19T08:40:45.610 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7b32f0700 1 -- 172.21.15.23:0/2236819929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ac10e720 con 0x7fc7ac0f5580 2021-11-19T08:40:45.610 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7a1ffb700 1 -- 172.21.15.23:0/2236819929 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7a800f040 con 0x7fc7ac0f5580 2021-11-19T08:40:45.610 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.619+0000 7fc7a1ffb700 1 -- 172.21.15.23:0/2236819929 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7a8014840 con 0x7fc7ac0f5580 2021-11-19T08:40:45.611 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.620+0000 7fc7b32f0700 1 -- 172.21.15.23:0/2236819929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac10ecb0 con 0x7fc7ac0f5580 2021-11-19T08:40:45.617 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.621+0000 7fc7a1ffb700 1 -- 172.21.15.23:0/2236819929 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc7a801b060 con 0x7fc7ac0f5580 2021-11-19T08:40:45.617 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.621+0000 7fc7a1ffb700 1 --2- 172.21.15.23:0/2236819929 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc79805c2a0 0x7fc79805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:45.618 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.621+0000 7fc7a1ffb700 1 -- 172.21.15.23:0/2236819929 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fc7a8075ee0 con 0x7fc7ac0f5580 2021-11-19T08:40:45.618 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.621+0000 7fc7b088b700 1 --2- 172.21.15.23:0/2236819929 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc79805c2a0 0x7fc79805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:45.618 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.622+0000 7fc7b088b700 1 --2- 172.21.15.23:0/2236819929 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc79805c2a0 0x7fc79805e760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc79c004e30 tx=0x7fc79c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:45.618 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.623+0000 7fc7c4e9bb80 1 Processor -- start 2021-11-19T08:40:45.618 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.623+0000 7fc7c4e9bb80 1 -- start start 2021-11-19T08:40:45.619 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.623+0000 7fc7c4e9bb80 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a4c9c7d1d0 0x55a4c9e0faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:45.619 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.623+0000 7fc7c4e9bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c9e0ffe0 0x55a4c9e10d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:45.619 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.623+0000 7fc7c4e9bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c9d4e4d0 con 0x55a4c9e0ffe0 2021-11-19T08:40:45.619 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.623+0000 7fc7c4e9bb80 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c9d623e0 con 0x55a4c9c7d1d0 2021-11-19T08:40:45.620 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.623+0000 7fc7b088b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c9e0ffe0 0x55a4c9e10d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:45.620 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.623+0000 7fc7b088b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c9e0ffe0 0x55a4c9e10d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47222/0 (socket says 172.21.15.23:47222) 2021-11-19T08:40:45.620 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.623+0000 7fc7b088b700 1 -- 172.21.15.23:0/2034501879 learned_addr learned my addr 172.21.15.23:0/2034501879 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:45.620 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.624+0000 7fc7b108c700 1 --2- 172.21.15.23:0/2034501879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a4c9c7d1d0 0x55a4c9e0faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:45.620 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.624+0000 7fc7b088b700 1 -- 172.21.15.23:0/2034501879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a4c9c7d1d0 msgr2=0x55a4c9e0faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-19T08:40:45.621 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.624+0000 7fc7b088b700 1 --2- 172.21.15.23:0/2034501879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a4c9c7d1d0 0x55a4c9e0faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.621 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.624+0000 7fc7b088b700 1 -- 172.21.15.23:0/2034501879 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a4c9dde3c0 con 0x55a4c9e0ffe0 2021-11-19T08:40:45.621 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.624+0000 7fc7b088b700 1 --2- 172.21.15.23:0/2034501879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c9e0ffe0 0x55a4c9e10d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fc79c011000 tx=0x7fc79c01c000).ready entity=mon.0 client_cookie=8c7179c32a1e56f8 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:45.621 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.624+0000 7fc796ffd700 1 -- 172.21.15.23:0/2034501879 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc79c024070 con 0x55a4c9e0ffe0 2021-11-19T08:40:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.624+0000 7fc796ffd700 1 -- 172.21.15.23:0/2034501879 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc79c019040 con 0x55a4c9e0ffe0 2021-11-19T08:40:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.624+0000 7fc796ffd700 1 -- 172.21.15.23:0/2034501879 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc79c00e040 con 0x55a4c9e0ffe0 2021-11-19T08:40:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.624+0000 7fc7c4e9bb80 1 -- 172.21.15.23:0/2034501879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c9e0ffe0 msgr2=0x55a4c9e10d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:40:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.624+0000 7fc7c4e9bb80 1 --2- 172.21.15.23:0/2034501879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c9e0ffe0 0x55a4c9e10d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fc79c011000 tx=0x7fc79c01c000).stop 2021-11-19T08:40:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7c4e9bb80 1 -- 172.21.15.23:0/2034501879 shutdown_connections 2021-11-19T08:40:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7c4e9bb80 1 --2- 172.21.15.23:0/2034501879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c9e0ffe0 0x55a4c9e10d40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7c4e9bb80 1 --2- 172.21.15.23:0/2034501879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a4c9c7d1d0 0x55a4c9e0faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7c4e9bb80 1 -- 172.21.15.23:0/2034501879 >> 172.21.15.23:0/2034501879 conn(0x55a4c9dd59b0 msgr2=0x55a4c9e0afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:40:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7c4e9bb80 1 -- 172.21.15.23:0/2034501879 shutdown_connections 2021-11-19T08:40:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7c4e9bb80 1 -- 172.21.15.23:0/2034501879 wait complete. 2021-11-19T08:40:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7c4e9bb80 1 -- start start 2021-11-19T08:40:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7c4e9bb80 1 Processor -- start 2021-11-19T08:40:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7c4e9bb80 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a4c9c7d1d0 0x55a4c9e18760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:45.625 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7c4e9bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c9e0ffe0 0x55a4c9e18c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:45.625 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7c4e9bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c9d62520 con 0x55a4c9e0ffe0 2021-11-19T08:40:45.625 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7c4e9bb80 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c9d62aa0 con 0x55a4c9c7d1d0 2021-11-19T08:40:45.625 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7c4e9bb80 10 client.0 ll_register_callbacks cb 0x55a4c9dcf700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2021-11-19T08:40:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7b088b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c9e0ffe0 0x55a4c9e18c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.625+0000 7fc7b088b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c9e0ffe0 0x55a4c9e18c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47226/0 (socket says 172.21.15.23:47226) 2021-11-19T08:40:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc7b088b700 1 -- 172.21.15.23:0/3267137691 learned_addr learned my addr 172.21.15.23:0/3267137691 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:40:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc7b088b700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a4c9c7d1d0 msgr2=0x55a4c9e18760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-11-19T08:40:45.627 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a4c9c7d1d0 0x55a4c9e18760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:45.627 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc7b088b700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a4c9c7d1d0 0x55a4c9e18760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:40:45.627 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc7b088b700 1 -- 172.21.15.23:0/3267137691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac0f7300 con 0x55a4c9e0ffe0 2021-11-19T08:40:45.628 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc7b088b700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c9e0ffe0 0x55a4c9e18c50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fc79c013490 tx=0x7fc79c01c000).ready entity=mon.0 client_cookie=9c18415d8877a5f server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:40:45.628 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc78bfff700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2021-11-19T08:40:45.628 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc78bfff700 1 -- 172.21.15.23:0/3267137691 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc79c024040 con 0x55a4c9e0ffe0 2021-11-19T08:40:45.628 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc7c4e9bb80 1 -- 172.21.15.23:0/3267137691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a4c9dddce0 con 0x55a4c9e0ffe0 2021-11-19T08:40:45.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc78bfff700 1 -- 172.21.15.23:0/3267137691 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fc79c01d8c0 con 0x55a4c9e0ffe0 2021-11-19T08:40:45.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc78bfff700 1 -- 172.21.15.23:0/3267137691 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc79c029420 con 0x55a4c9e0ffe0 2021-11-19T08:40:45.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc7c4e9bb80 10 client.14556 fetch_fsmap learned FSMap version 14 2021-11-19T08:40:45.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc7c4e9bb80 1 -- 172.21.15.23:0/3267137691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=3+}) v3 -- 0x55a4c9d51630 con 0x55a4c9e0ffe0 2021-11-19T08:40:45.630 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc78bfff700 1 -- 172.21.15.23:0/3267137691 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x55a4c9dddce0 con 0x55a4c9e0ffe0 2021-11-19T08:40:45.630 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc78bfff700 1 -- 172.21.15.23:0/3267137691 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 14) v1 ==== 1712+0+0 (secure 0 0 0) 0x7fc79c0283d0 con 0x55a4c9e0ffe0 2021-11-19T08:40:45.631 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc7c4e9bb80 10 client.14556 fetch_fsmap finished waiting for FSMap version 14 2021-11-19T08:40:45.631 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc7c4e9bb80 10 client.14556 resolve_mds: resolved name 'cephfs.smithi023.fuvzkr' to daemon mds.cephfs.smithi023.fuvzkr 2021-11-19T08:40:45.631 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc7c4e9bb80 20 client.14556 populate_metadata read hostname 'smithi023' 2021-11-19T08:40:45.631 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.626+0000 7fc7c4e9bb80 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-11-19T08:40:45.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.627+0000 7fc7c4e9bb80 4 client.14556 mds_command: new command op to 14434 tid=0[{"prefix": "get_command_descriptions"}] 2021-11-19T08:40:45.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.627+0000 7fc7c4e9bb80 1 -- 172.21.15.23:0/3267137691 --> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a4c9d58df0 con 0x55a4c9e214a0 2021-11-19T08:40:45.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.627+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:6826/530484646 2021-11-19T08:40:45.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.627+0000 7fc78bfff700 1 client.14556 ms_handle_refused on v2:172.21.15.23:6826/530484646 2021-11-19T08:40:45.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.627+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-11-19T08:40:45.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:45 smithi134 conmon[36363]: audit 2021-11-19T08:40:44.520190+0000 mon.smithi023 (mon.0) 544 : audit [DBG] from='client.? 172.21.15.23:0/296349821' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-19T08:40:45.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:45 smithi134 conmon[36363]: audit 2021-11-19T08:40:44.832223+0000 mon.smithi023 ( 2021-11-19T08:40:45.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:45 smithi134 conmon[36363]: mon.0) 545 : audit [DBG] from='client.? 172.21.15.23:0/1532453009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-19T08:40:45.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:45 smithi134 conmon[36363]: audit 2021-11-19T08:40:45.141177+0000 mon.smithi023 (mon.0) 546 : audit [DBG] from='client.? 172.21.15.23:0/3521975289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-19T08:40:45.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:45 smithi134 conmon[36363]: audit 2021-11-19T08:40:45.484569+0000 mon.smithi023 (mon.0) 547 : audit [DBG] from='client.? 172.21.15.23:0/2115787125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:40:45.812 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.828+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:6826/530484646 2021-11-19T08:40:45.813 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.828+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-11-19T08:40:45.813 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:45.828+0000 7fc78bfff700 1 client.14556 ms_handle_refused on v2:172.21.15.23:6826/530484646 2021-11-19T08:40:45.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:45 smithi023 conmon[33490]: audit 2021-11-19T08:40:44.520190+0000 2021-11-19T08:40:45.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:45 smithi023 conmon[33490]: mon.smithi023 (mon.0) 544 : audit [DBG] from='client.? 172.21.15.23:0/296349821' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-11-19T08:40:45.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:45 smithi023 conmon[33490]: audit 2021-11-19T08:40:44.832223+0000 mon.smithi023 (mon.0) 545 : audit [DBG] from='client.? 172.21.15.23:0/1532453009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2021-11-19T08:40:45.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:45 smithi023 conmon[33490]: audit 2021-11-19T08:40:45.141177+0000 mon.smithi023 (mon.0) 546 : audit [DBG] from='client.? 172.21.15.23:0/3521975289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2021-11-19T08:40:45.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:45 smithi023 conmon[33490]: audit 2021-11-19T08:40:45.484569+0000 mon.smithi023 (mon.0) 547 : audit [DBG] from='client.? 172.21.15.23:0/2115787125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:40:46.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:46.229+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:6826/530484646 2021-11-19T08:40:46.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:46.229+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-11-19T08:40:46.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:46.229+0000 7fc78bfff700 1 client.14556 ms_handle_refused on v2:172.21.15.23:6826/530484646 2021-11-19T08:40:46.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:46 smithi134 conmon[36363]: cluster 2021-11-19T08:40:44.830080+0000 mgr.smithi023.ehlpcl (mgr.14164) 202 : cluster [DBG] pgmap v163: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 7 op/s 2021-11-19T08:40:46.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:46 smithi023 conmon[33490]: cluster 2021-11-19T08:40:44.830080+0000 mgr.smithi023.ehlpcl (mgr.14164) 202 : cluster 2021-11-19T08:40:46.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:46 smithi023 conmon[33490]: [DBG] pgmap v163: 65 pgs: 65 active+clean; 3.8 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 7 op/s 2021-11-19T08:40:47.015 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:47.030+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:6826/530484646 2021-11-19T08:40:47.015 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:47.030+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-11-19T08:40:47.015 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:47.030+0000 7fc78bfff700 1 client.14556 ms_handle_refused on v2:172.21.15.23:6826/530484646 2021-11-19T08:40:48.616 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:48.631+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:6826/530484646 2021-11-19T08:40:48.616 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:48.631+0000 7fc78bfff700 1 client.14556 ms_handle_refused on v2:172.21.15.23:6826/530484646 2021-11-19T08:40:48.616 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:48.631+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2021-11-19T08:40:48.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:48 smithi023 conmon[33490]: cluster 2021-11-19T08:40:46.830814+0000 mgr.smithi023.ehlpcl (mgr.14164) 203 : cluster [DBG] pgmap v164: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 3.1 KiB/s rd, 341 B/s wr, 6 op/s 2021-11-19T08:40:48.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:48 smithi134 conmon[36363]: cluster 2021-11-19T08:40:46.830814+0000 mgr.smithi023.ehlpcl (mgr.14164) 203 : cluster [DBG] pgmap v164: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 3.1 KiB/s rd, 341 B/s wr, 6 op/s 2021-11-19T08:40:48.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:48 smithi134 conmon[36363]: 2021-11-19T08:40:50.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:50 smithi134 conmon[36363]: cluster 2021-11-19T08:40:48.831276+0000 mgr.smithi023.ehlpcl (mgr.14164) 204 : cluster [DBG] pgmap v165: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2021-11-19T08:40:50.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:50 smithi023 conmon[33490]: cluster 2021-11-19T08:40:48.831276+0000 mgr.smithi023.ehlpcl (mgr.14164) 204 : cluster [DBG] pgmap v165: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2021-11-19T08:40:51.817 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:51.832+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:6826/530484646 2021-11-19T08:40:51.817 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:51.832+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 6.400000 2021-11-19T08:40:51.817 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:51.832+0000 7fc78bfff700 1 client.14556 ms_handle_refused on v2:172.21.15.23:6826/530484646 2021-11-19T08:40:52.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:52 smithi134 conmon[36363]: cluster 2021-11-19T08:40:50.831992+0000 mgr.smithi023.ehlpcl (mgr.14164) 205 : cluster [DBG] pgmap v166: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 5 op/s 2021-11-19T08:40:52.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:52 smithi023 conmon[33490]: cluster 2021-11-19T08:40:50.831992+0000 mgr.smithi023.ehlpcl (mgr.14164) 205 : cluster 2021-11-19T08:40:52.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:52 smithi023 conmon[33490]: [DBG] pgmap v166: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 5 op/s 2021-11-19T08:40:54.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:54 smithi134 conmon[36363]: cluster 2021-11-19T08:40:52.832442+0000 mgr.smithi023.ehlpcl (mgr.14164) 206 : cluster [DBG] pgmap v167: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 4 op/s 2021-11-19T08:40:54.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:54 smithi023 conmon[33490]: cluster 2021-11-19T08:40:52.832442+0000 mgr.smithi023.ehlpcl (mgr.14164) 206 : cluster 2021-11-19T08:40:54.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:54 smithi023 conmon[33490]: [DBG] pgmap v167: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 4 op/s 2021-11-19T08:40:55.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:55 smithi134 conmon[36363]: cluster 2021-11-19T08:40:55.187237+0000 mon.smithi023 (mon.0) 548 : cluster [INF] daemon mds.cephfs.smithi023.fuvzkr restarted 2021-11-19T08:40:55.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:55 smithi134 conmon[36363]: cluster 2021-11-19T08:40:55.191086+0000 mon.smithi023 (mon.0) 549 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2021-11-19T08:40:55.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:55 smithi023 conmon[33490]: cluster 2021-11-19T08:40:55.187237+0000 mon.smithi023 (mon.0) 548 : cluster [INF] daemon mds.cephfs.smithi023.fuvzkr restarted 2021-11-19T08:40:56.029 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:55 smithi023 conmon[33490]: cluster 2021-11-19T08:40:55.191086+0000 mon.smithi023 ( 2021-11-19T08:40:56.029 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:55 smithi023 conmon[33490]: mon.0) 549 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2021-11-19T08:40:56.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: cluster 2021-11-19T08:40:54.832945+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:40:56.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: 207 : cluster [DBG] pgmap v168: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 255 B/s wr, 4 op/s 2021-11-19T08:40:56.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: cluster 2021-11-19T08:40:55.531462+0000 mon.smithi023 (mon.0 2021-11-19T08:40:56.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: ) 550 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2021-11-19T08:40:56.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: cluster 2021-11 2021-11-19T08:40:56.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: -19T08:40:55.541781+0000 mon.smithi023 (mon 2021-11-19T08:40:56.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: .0) 551 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3054538148,v1:172.21.15.23:6827/3054538148] up:boot 2021-11-19T08:40:56.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: cluster 2021-11-19T08:40:56.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: 2021-11-19T08:40:55.541853+0000 2021-11-19T08:40:56.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: mon.smithi023 (mon.0) 552 : cluster 2021-11-19T08:40:56.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: [INF] Standby daemon mds.cephfs.smithi134.jqsnzo assigned to filesystem cephfs as rank 0 2021-11-19T08:40:56.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: cluster 2021-11-19T08:40:56.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: 2021-11-19T08:40 2021-11-19T08:40:56.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: :55.544229+0000 mon.smithi023 (mon 2021-11-19T08:40:56.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: .0) 553 : cluster [DBG] 2021-11-19T08:40:56.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: fsmap cephfs:1/2 {1=cephfs.smithi134.opxllb=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2021-11-19T08:40:56.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: audit 2021-11-19 2021-11-19T08:40:56.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: T08:40:55.544341+0000 2021-11-19T08:40:56.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: mon.smithi023 (mon.0) 554 : audit 2021-11-19T08:40:56.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.fuvzkr"}]: dispatch 2021-11-19T08:40:56.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: cluster 2021-11-19T08:40:56.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: 2021-11-19T08:40 2021-11-19T08:40:56.795 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: :55.551228+0000 mon.smithi023 (mon 2021-11-19T08:40:56.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: .0) 555 : cluster [DBG] 2021-11-19T08:40:56.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:56 smithi134 conmon[36363]: fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:replay,1=cephfs.smithi134.opxllb=up:active} 1 up:standby-replay 1 up:standby 2021-11-19T08:40:56.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:56 smithi023 conmon[33490]: cluster 2021-11-19T08:40:54.832945+0000 mgr.smithi023.ehlpcl (mgr.14164) 207 : cluster [DBG] pgmap v168: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 255 B/s wr, 4 op/s 2021-11-19T08:40:56.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:56 smithi023 conmon[33490]: cluster 2021-11-19T08:40:55.531462+0000 mon.smithi023 (mon.0) 550 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2021-11-19T08:40:56.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:56 smithi023 conmon[33490]: cluster 2021-11-19T08:40:55.541781+0000 mon.smithi023 (mon.0) 551 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3054538148,v1:172.21.15.23:6827/3054538148] up:boot 2021-11-19T08:40:56.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:56 smithi023 conmon[33490]: cluster 2021-11-19T08:40:55.541853+0000 mon.smithi023 (mon.0) 552 : cluster [INF] Standby daemon mds.cephfs.smithi134.jqsnzo assigned to filesystem cephfs as rank 0 2021-11-19T08:40:56.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:56 smithi023 conmon[33490]: cluster 2021-11-19T08:40:55.544229+0000 mon.smithi023 (mon.0) 553 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi134.opxllb=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2021-11-19T08:40:56.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:56 smithi023 conmon[33490]: audit 2021-11-19T08:40:55.544341+0000 mon.smithi023 (mon. 2021-11-19T08:40:56.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:56 smithi023 conmon[33490]: 0) 554 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.fuvzkr"}]: dispatch 2021-11-19T08:40:56.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:56 smithi023 conmon[33490]: cluster 2021- 2021-11-19T08:40:56.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:56 smithi023 conmon[33490]: 11-19T08:40:55.551228+0000 2021-11-19T08:40:56.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:56 smithi023 conmon[33490]: mon.smithi023 (mon.0) 555 : cluster 2021-11-19T08:40:56.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:56 smithi023 conmon[33490]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:replay,1=cephfs.smithi134.opxllb=up:active} 1 up:standby-replay 1 up:standby 2021-11-19T08:40:57.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:57 smithi023 conmon[33490]: cluster 2021-11-19T08:40:56.556234+0000 mon.smithi023 (mon.0) 556 : cluster [DBG] mds.? [v2:172.21.15.134:6824/1874302373,v1:172.21.15.134:6825/1874302373] up:resolve 2021-11-19T08:40:57.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:57 smithi023 conmon[33490]: cluster 2021-11-19T08:40:56.556308+0000 mon.smithi023 (mon.0) 557 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi134.opxllb=up:active} 1 up:standby-replay 1 up:standby 2021-11-19T08:40:58.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:57 smithi134 conmon[36363]: cluster 2021-11-19T08:40:56.556234+0000 mon.smithi023 (mon.0) 556 : cluster [DBG] 2021-11-19T08:40:58.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:57 smithi134 conmon[36363]: mds.? [v2:172.21.15.134:6824/1874302373,v1:172.21.15.134:6825/1874302373] up:resolve 2021-11-19T08:40:58.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:57 smithi134 conmon[36363]: cluster 2021-11-19T08:40:56.556308+0000 mon.smithi023 (mon.0) 557 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi134.opxllb=up:active} 1 up:standby-replay 1 up:standby 2021-11-19T08:40:58.224 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:58.239+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:40:58.224 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:58.239+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-19T08:40:58.224 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:58.239+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-19T08:40:58.225 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:58.239+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-19T08:40:58.225 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:40:58.239+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 12.800000 2021-11-19T08:40:58.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:58 smithi023 conmon[33490]: cluster 2021-11-19T08:40:56.833653+0000 mgr.smithi023.ehlpcl (mgr.14164) 208 : cluster [DBG] 2021-11-19T08:40:58.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:40:58 smithi023 conmon[33490]: pgmap v170: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2021-11-19T08:40:59.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:58 smithi134 conmon[36363]: cluster 2021-11-19T08:40:56.833653+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:40:59.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:40:58 smithi134 conmon[36363]: .14164) 208 : cluster [DBG] pgmap v170: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2021-11-19T08:41:00.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:00 smithi023 conmon[33490]: cluster 2021-11-19T08:40:58.834271+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:41:00.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:00 smithi023 conmon[33490]: 209 : cluster [DBG] pgmap v171: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2021-11-19T08:41:00.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:00 smithi023 conmon[33490]: cluster 2021-11-19T08:41:00.188640+0000 mon.smithi023 (mon.0) 558 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3054538148,v1:172.21.15.23:6827/3054538148] up:standby 2021-11-19T08:41:00.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:00 smithi023 conmon[33490]: cluster 2021- 2021-11-19T08:41:00.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:00 smithi023 conmon[33490]: 11-19T08:41:00.188721+0000 2021-11-19T08:41:00.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:00 smithi023 conmon[33490]: mon.smithi023 (mon.0) 559 : cluster 2021-11-19T08:41:00.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:00 smithi023 conmon[33490]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi134.opxllb=up:active} 1 up:standby-replay 1 up:standby 2021-11-19T08:41:01.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:00 smithi134 conmon[36363]: cluster 2021-11-19T08:40:58.834271+0000 mgr.smithi023.ehlpcl (mgr.14164) 209 : cluster 2021-11-19T08:41:01.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:00 smithi134 conmon[36363]: [DBG] pgmap v171: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2021-11-19T08:41:01.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:00 smithi134 conmon[36363]: cluster 2021-11-19T08:41:00.188640+0000 mon.smithi023 (mon.0) 558 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3054538148,v1:172.21.15.23:6827/3054538148] up:standby 2021-11-19T08:41:01.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:00 smithi134 conmon[36363]: cluster 2021-11-19T08:41:00.188721+0000 2021-11-19T08:41:01.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:00 smithi134 conmon[36363]: mon.smithi023 (mon.0) 559 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi134.opxllb=up:active} 1 up:standby-replay 1 up:standby 2021-11-19T08:41:02.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:02 smithi023 conmon[33490]: cluster 2021-11-19T08:41:02.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:02 smithi023 conmon[33490]: 2021-11-19T08:41:00.835022+0000 mgr.smithi023.ehlpcl (mgr.14164) 210 : cluster [DBG] pgmap v172: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2021-11-19T08:41:03.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:02 smithi134 conmon[36363]: cluster 2021-11-19T08:41:00.835022+0000 mgr.smithi023.ehlpcl (mgr.14164) 210 : cluster [DBG] pgmap v172: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2021-11-19T08:41:04.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:04 smithi023 conmon[33490]: cluster 2021-11-19T08:41:02.835531+0000 mgr.smithi023.ehlpcl (mgr.14164) 211 : cluster [DBG] pgmap v173: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2021-11-19T08:41:05.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:04 smithi134 conmon[36363]: cluster 2021-11-19T08:41:02.835531+0000 mgr.smithi023.ehlpcl (mgr.14164) 211 : 2021-11-19T08:41:05.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:04 smithi134 conmon[36363]: cluster [DBG] pgmap v173: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2021-11-19T08:41:06.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:06 smithi023 conmon[33490]: cluster 2021-11-19T08:41:04.836124+0000 mgr.smithi023.ehlpcl (mgr.14164) 212 2021-11-19T08:41:06.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:06 smithi023 conmon[33490]: : cluster [DBG] pgmap v174: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2021-11-19T08:41:07.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:06 smithi134 conmon[36363]: cluster 2021-11-19T08:41:04.836124+0000 mgr.smithi023.ehlpcl (mgr.14164) 212 : 2021-11-19T08:41:07.018 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:06 smithi134 conmon[36363]: cluster [DBG] pgmap v174: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2021-11-19T08:41:08.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:08 smithi023 conmon[33490]: cluster 2021-11-19T08:41:06.836855+0000 mgr.smithi023.ehlpcl (mgr.14164) 213 : cluster [DBG] pgmap v175: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2021-11-19T08:41:09.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:08 smithi134 conmon[36363]: cluster 2021-11-19T08:41:06.836855+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:41:09.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:08 smithi134 conmon[36363]: 14164) 213 : cluster [DBG] pgmap v175: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2021-11-19T08:41:10.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:10 smithi023 conmon[33490]: cluster 2021-11-19T08:41:08.837313+0000 mgr.smithi023.ehlpcl (mgr.14164) 214 2021-11-19T08:41:10.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:10 smithi023 conmon[33490]: : cluster [DBG] pgmap v176: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2021-11-19T08:41:11.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:11.040+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:41:11.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:11.040+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-19T08:41:11.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:11.040+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-19T08:41:11.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:11.040+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-19T08:41:11.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:11.040+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-19T08:41:11.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:10 smithi134 conmon[36363]: cluster 2021-11-19T08:41:08.837313+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:41:11.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:10 smithi134 conmon[36363]: 214 : cluster [DBG] pgmap v176: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2021-11-19T08:41:11.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:11 smithi023 conmon[33490]: cluster 2021-11-19T08:41:11.286914+0000 mon.smithi023 (mon. 2021-11-19T08:41:11.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:11 smithi023 conmon[33490]: 0) 560 : cluster [INF] daemon mds.cephfs.smithi134.opxllb restarted 2021-11-19T08:41:11.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:11 smithi023 conmon[33490]: cluster 2021-11-19T08:41:11.296439+0000 mon.smithi023 (mon.0) 561 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2021-11-19T08:41:12.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:11 smithi134 conmon[36363]: cluster 2021-11-19T08:41:11.286914+0000 mon.smithi023 (mon.0) 2021-11-19T08:41:12.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:11 smithi134 conmon[36363]: 560 : cluster [INF] daemon mds.cephfs.smithi134.opxllb restarted 2021-11-19T08:41:12.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:11 smithi134 conmon[36363]: cluster 2021-11-19T08:41:11.296439+0000 mon.smithi023 (mon.0) 561 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2021-11-19T08:41:12.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:12 smithi023 conmon[33490]: cluster 2021-11-19T08:41:10.837940+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:41:12.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:12 smithi023 conmon[33490]: 215 : cluster [DBG] pgmap v177: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2021-11-19T08:41:12.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:12 smithi023 conmon[33490]: cluster 2021-11-19T08:41:11.621711+0000 mon.smithi023 (mon.0) 562 : cluster [DBG] mds.? [v2:172.21.15.134:6826/954232334,v1:172.21.15.134:6827/954232334] up:boot 2021-11-19T08:41:12.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:12 smithi023 conmon[33490]: cluster 2021 2021-11-19T08:41:12.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:12 smithi023 conmon[33490]: -11-19T08:41:11.621786+0000 mon.smithi023 (mon.0) 563 : cluster [INF] Standby daemon mds.cephfs.smithi023.wazanl assigned to filesystem cephfs as rank 1 2021-11-19T08:41:12.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:12 smithi023 conmon[33490]: cluster 2021-11-19T08:41:11 2021-11-19T08:41:12.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:12 smithi023 conmon[33490]: .622108+0000 mon.smithi023 (mon.0) 564 : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi134.jqsnzo=up:resolve} 1 up:standby-replay 2 up:standby, 1 failed 2021-11-19T08:41:12.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:12 smithi023 conmon[33490]: audit 2021-11-19T08:41:11.622242+0000 mon.smithi023 (mon.0) 565 : 2021-11-19T08:41:12.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:12 smithi023 conmon[33490]: audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.opxllb"}]: dispatch 2021-11-19T08:41:12.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:12 smithi023 conmon[33490]: cluster 2021-11-19T08:41:12.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:12 smithi023 conmon[33490]: 2021-11-19T08:41:11.624899+0000 mon.smithi023 (mon 2021-11-19T08:41:12.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:12 smithi023 conmon[33490]: .0) 566 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.wazanl=up:replay} 2 up:standby 2021-11-19T08:41:13.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: cluster 2021-11-19T08:41:10.837940+0000 2021-11-19T08:41:13.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: mgr.smithi023.ehlpcl (mgr.14164) 215 : cluster [DBG] pgmap v177: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2021-11-19T08:41:13.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: cluster 2021-11-19T08 2021-11-19T08:41:13.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: :41:11.621711+0000 mon.smithi023 (mon.0) 562 : cluster 2021-11-19T08:41:13.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: [DBG] mds.? [v2:172.21.15.134:6826/954232334,v1:172.21.15.134:6827/954232334] up:boot 2021-11-19T08:41:13.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: cluster 2021- 2021-11-19T08:41:13.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: 11-19T08:41:11.621786 2021-11-19T08:41:13.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 563 : 2021-11-19T08:41:13.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: cluster [INF] Standby daemon mds.cephfs.smithi023.wazanl assigned to filesystem cephfs as rank 1 2021-11-19T08:41:13.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: cluster 2021-11-19T08:41:13.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: 2021-11-19T08:41 2021-11-19T08:41:13.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: :11.622108+0000 mon.smithi023 (mon 2021-11-19T08:41:13.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: .0) 564 : cluster [DBG] 2021-11-19T08:41:13.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: fsmap cephfs:1/2 {0=cephfs.smithi134.jqsnzo=up:resolve} 1 up:standby-replay 2 up:standby, 1 failed 2021-11-19T08:41:13.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:41:13.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: 11-19T08:41:11.622242 2021-11-19T08:41:13.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 565 2021-11-19T08:41:13.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.opxllb"}]: dispatch 2021-11-19T08:41:13.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: cluster 2021 2021-11-19T08:41:13.045 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: -11-19T08:41:11. 2021-11-19T08:41:13.046 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: 624899+0000 mon.smithi023 (mon.0) 2021-11-19T08:41:13.046 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:12 smithi134 conmon[36363]: 566 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.wazanl=up:replay} 2 up:standby 2021-11-19T08:41:13.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:13 smithi023 conmon[33490]: cluster 2021-11-19T08:41:12.628927+0000 mon.smithi023 (mon. 2021-11-19T08:41:13.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:13 smithi023 conmon[33490]: 0) 567 : cluster [DBG] mds.? [v2:172.21.15.23:6828/2176063367,v1:172.21.15.23:6829/2176063367] up:resolve 2021-11-19T08:41:13.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:13 smithi023 conmon[33490]: cluster 2021-11-19T08:41:12.629006+0000 mon.smithi023 (mon.0) 568 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.wazanl=up:resolve} 2 up:standby 2021-11-19T08:41:14.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:13 smithi134 conmon[36363]: cluster 2021-11-19T08:41:12.628927+0000 mon.smithi023 (mon.0) 567 : cluster [DBG] 2021-11-19T08:41:14.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:13 smithi134 conmon[36363]: mds.? [v2:172.21.15.23:6828/2176063367,v1:172.21.15.23:6829/2176063367] up:resolve 2021-11-19T08:41:14.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:13 smithi134 conmon[36363]: cluster 2021-11-19T08:41:12.629006+0000 mon.smithi023 (mon.0) 568 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.wazanl=up:resolve} 2 up:standby 2021-11-19T08:41:14.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:14 smithi023 conmon[33490]: cluster 2021-11-19T08:41:12.838414+0000 mgr.smithi023.ehlpcl 2021-11-19T08:41:14.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:14 smithi023 conmon[33490]: (mgr.14164) 216 : cluster [DBG] pgmap v179: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2021-11-19T08:41:14.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:14 smithi023 conmon[33490]: cluster 2021-11-19T08:41:13.636236+0000 mon.smithi023 (mon.0) 569 : cluster [DBG] mds.? [v2:172.21.15.23:6828/2176063367,v1:172.21.15.23:6829/2176063367] up:reconnect 2021-11-19T08:41:14.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:14 smithi023 conmon[33490]: 2021-11-19T08:41:14.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:14 smithi023 conmon[33490]: cluster 2021-11-19T08:41:13.636318+0000 mon.smithi023 2021-11-19T08:41:14.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:14 smithi023 conmon[33490]: (mon.0) 570 : cluster [DBG] mds.? [v2:172.21.15.134:6824/1874302373,v1:172.21.15.134:6825/1874302373] up:reconnect 2021-11-19T08:41:14.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:14 smithi023 conmon[33490]: cluster 2021-11-19T08:41:13.636405+0000 2021-11-19T08:41:14.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:14 smithi023 conmon[33490]: mon.smithi023 (mon.0) 571 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:reconnect,1=cephfs.smithi023.wazanl=up:reconnect} 2 up:standby 2021-11-19T08:41:14.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:14 smithi023 conmon[33490]: cluster 2021-11-19T08:41: 2021-11-19T08:41:14.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:14 smithi023 conmon[33490]: 13.659786+0000 mds.cephfs.smithi023.wazanl (mds.1) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249998 2021-11-19T08:41:14.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:14 smithi023 conmon[33490]: cluster 2021-11- 2021-11-19T08:41:14.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:14 smithi023 conmon[33490]: 19T08:41:13.660574+0000 mds.cephfs.smithi134.jqsnzo (mds.0) 1 2021-11-19T08:41:14.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:14 smithi023 conmon[33490]: : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249994 2021-11-19T08:41:15.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:14 smithi134 conmon[36363]: cluster 2021-11-19T08:41:12.838414+0000 mgr.smithi023.ehlpcl (mgr.14164) 216 : cluster [DBG] 2021-11-19T08:41:15.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:14 smithi134 conmon[36363]: pgmap v179: 65 pgs: 65 active+clean; 4.1 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2021-11-19T08:41:15.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:14 smithi134 conmon[36363]: cluster 2021-11-19T08:41:13.636236+0000 mon.smithi023 (mon.0) 569 : cluster [DBG] mds.? [v2:172.21.15.23:6828/2176063367,v1:172.21.15.23:6829/2176063367] up:reconnect 2021-11-19T08:41:15.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:14 smithi134 conmon[36363]: cluster 2021-11-19T08:41:13.636318+0000 mon.smithi023 (mon.0 2021-11-19T08:41:15.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:14 smithi134 conmon[36363]: ) 570 : cluster [DBG] mds.? [v2:172.21.15.134:6824/1874302373,v1:172.21.15.134:6825/1874302373] up:reconnect 2021-11-19T08:41:15.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:14 smithi134 conmon[36363]: cluster 2021-11-19T08:41: 2021-11-19T08:41:15.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:14 smithi134 conmon[36363]: 13.636405+0000 mon.smithi023 (mon.0) 571 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:reconnect,1=cephfs.smithi023.wazanl=up:reconnect} 2 up:standby 2021-11-19T08:41:15.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:14 smithi134 conmon[36363]: cluster 2021-11- 2021-11-19T08:41:15.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:14 smithi134 conmon[36363]: 19T08:41:13.659786+0000 mds.cephfs.smithi023.wazanl (mds.1 2021-11-19T08:41:15.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:14 smithi134 conmon[36363]: ) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249998 2021-11-19T08:41:15.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:14 smithi134 conmon[36363]: cluster 2021-11 2021-11-19T08:41:15.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:14 smithi134 conmon[36363]: -19T08:41:13.660574+0000 mds.cephfs.smithi134.jqsnzo ( 2021-11-19T08:41:15.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:14 smithi134 conmon[36363]: mds.0) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249994 2021-11-19T08:41:15.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:15 smithi023 conmon[33490]: cluster 2021-11-19T08:41:14. 2021-11-19T08:41:15.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:15 smithi023 conmon[33490]: 639691+0000 mon.smithi023 (mon.0) 572 : cluster [DBG] mds.? [v2:172.21.15.23:6828/2176063367,v1:172.21.15.23:6829/2176063367] up:rejoin 2021-11-19T08:41:15.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:15 smithi023 conmon[33490]: cluster 2021-11-19T08:41:14.639760+0000 2021-11-19T08:41:15.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:15 smithi023 conmon[33490]: mon.smithi023 (mon.0) 573 : cluster [DBG] mds.? [v2:172.21.15.134:6824/1874302373,v1:172.21.15.134:6825/1874302373] up:rejoin 2021-11-19T08:41:15.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:15 smithi023 conmon[33490]: cluster 2021- 2021-11-19T08:41:15.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:15 smithi023 conmon[33490]: 11-19T08:41:14.639836+0000 mon.smithi023 2021-11-19T08:41:15.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:15 smithi023 conmon[33490]: (mon.0) 574 : cluster [DBG] 2021-11-19T08:41:15.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:15 smithi023 conmon[33490]: fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:rejoin,1=cephfs.smithi023.wazanl=up:rejoin} 2 up:standby 2021-11-19T08:41:15.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:15 smithi023 conmon[33490]: cluster 2021-11-19T08:41:15.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:15 smithi023 conmon[33490]: 2021-11-19T08:41:14.644525 2021-11-19T08:41:15.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:15 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 575 2021-11-19T08:41:15.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:15 smithi023 conmon[33490]: : cluster [INF] daemon mds.cephfs.smithi023.wazanl is now active in filesystem cephfs as rank 1 2021-11-19T08:41:16.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:15 smithi134 conmon[36363]: cluster 2021-11-19T08:41:14.639691+0000 mon.smithi023 (mon.0) 572 : cluster [DBG] mds.? [v2:172.21.15.23:6828/2176063367,v1:172.21.15.23:6829/2176063367] up:rejoin 2021-11-19T08:41:16.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:15 smithi134 conmon[36363]: cluster 2021-11-19T08:41:14.639760+0000 mon.smithi023 (mon.0) 573 : cluster [DBG] mds.? [v2:172.21.15.134:6824/1874302373,v1:172.21.15.134:6825/1874302373] up:rejoin 2021-11-19T08:41:16.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:15 smithi134 conmon[36363]: cluster 2021-11-19T08:41:14.639836 2021-11-19T08:41:16.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:15 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 574 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:rejoin,1=cephfs.smithi023.wazanl=up:rejoin} 2 up:standby 2021-11-19T08:41:16.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:15 smithi134 conmon[36363]: cluster 2021-11-19T08 2021-11-19T08:41:16.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:15 smithi134 conmon[36363]: :41:14.644525+0000 mon.smithi023 (mon.0) 575 : cluster [INF] daemon mds.cephfs.smithi023.wazanl is now active in filesystem cephfs as rank 1 2021-11-19T08:41:16.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:16 smithi023 conmon[33490]: cluster 2021-11-19T08:41:14.839093+0000 mgr.smithi023.ehlpcl (mgr.14164) 217 2021-11-19T08:41:16.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:16 smithi023 conmon[33490]: : cluster [DBG] pgmap v180: 65 pgs: 65 active+clean; 4.2 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 102 B/s wr, 2 op/s 2021-11-19T08:41:16.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:16 smithi023 conmon[33490]: cluster 2021-11-19T08:41:15.650828+0000 mon.smithi023 (mon.0) 576 : cluster [DBG] mds.? [v2:172.21.15.23:6828/2176063367,v1:172.21.15.23:6829/2176063367] up:active 2021-11-19T08:41:16.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:16 smithi023 conmon[33490]: cluster 2021-11-19T08:41:15.650900+0000 mon.smithi023 (mon.0) 577 : cluster [DBG] mds.? [v2:172.21.15.134:6824/1874302373,v1:172.21.15.134:6825/1874302373] up:clientreplay 2021-11-19T08:41:16.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:16 smithi023 conmon[33490]: cluster 2021-11-19T08:41:15.650982+0000 mon.smithi023 (mon.0) 578 : cluster [DBG] mds.? [v2:172.21.15.134:6826/954232334,v1:172.21.15.134:6827/954232334] up:standby 2021-11-19T08:41:16.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:16 smithi023 conmon[33490]: cluster 2021-11-19T08:41:15.651056+0000 mon.smithi023 (mon.0) 579 2021-11-19T08:41:16.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:16 smithi023 conmon[33490]: : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:clientreplay,1=cephfs.smithi023.wazanl=up:active} 2 up:standby 2021-11-19T08:41:17.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:16 smithi134 conmon[36363]: cluster 2021-11-19T08:41:14.839093+0000 mgr.smithi023.ehlpcl (mgr.14164) 217 : cluster [DBG] pgmap v180: 65 pgs: 65 active+clean; 4.2 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 102 B/s wr, 2 op/s 2021-11-19T08:41:17.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:16 smithi134 conmon[36363]: cluster 2021-11-19T08:41:15.650828+0000 mon.smithi023 (mon.0) 576 : cluster [DBG] mds.? [v2:172.21.15.23:6828/2176063367,v1:172.21.15.23:6829/2176063367] up:active 2021-11-19T08:41:17.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:16 smithi134 conmon[36363]: cluster 2021-11-19T08:41:15.650900+0000 2021-11-19T08:41:17.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:16 smithi134 conmon[36363]: mon.smithi023 (mon.0) 577 : cluster [DBG] mds.? [v2:172.21.15.134:6824/1874302373,v1:172.21.15.134:6825/1874302373] up:clientreplay 2021-11-19T08:41:17.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:16 smithi134 conmon[36363]: cluster 2021-11-19T08:41:17.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:16 smithi134 conmon[36363]: 2021-11-19T08:41:15.650982+0000 mon.smithi023 (mon.0) 2021-11-19T08:41:17.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:16 smithi134 conmon[36363]: 578 : cluster [DBG] mds.? [v2:172.21.15.134:6826/954232334,v1:172.21.15.134:6827/954232334] up:standby 2021-11-19T08:41:17.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:16 smithi134 conmon[36363]: cluster 2021-11-19 2021-11-19T08:41:17.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:16 smithi134 conmon[36363]: T08:41:15.651056+0000 mon.smithi023 (mon 2021-11-19T08:41:17.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:16 smithi134 conmon[36363]: .0) 579 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:clientreplay,1=cephfs.smithi023.wazanl=up:active} 2 up:standby 2021-11-19T08:41:18.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:18 smithi023 conmon[33490]: cluster 2021-11-19T08:41:16.839814+0000 mgr.smithi023.ehlpcl (mgr.14164) 218 : cluster [DBG] 2021-11-19T08:41:18.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:18 smithi023 conmon[33490]: pgmap v181: 65 pgs: 65 active+clean; 4.4 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 409 B/s wr, 2 op/s 2021-11-19T08:41:19.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:18 smithi134 conmon[36363]: cluster 2021-11-19T08:41:16.839814+0000 mgr.smithi023.ehlpcl (mgr.14164) 218 : cluster [DBG] pgmap v181: 65 pgs: 65 active+clean; 4.4 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 409 B/s wr, 2 op/s 2021-11-19T08:41:21.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:20 smithi134 conmon[36363]: cluster 2021-11-19T08:41:18.840310+0000 mgr.smithi023.ehlpcl (mgr.14164) 219 : cluster [DBG] pgmap v182: 65 pgs: 65 active+clean; 4.4 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 409 B/s wr, 2 op/s 2021-11-19T08:41:21.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:20 smithi023 conmon[33490]: cluster 2021-11-19T08:41:18.840310+0000 mgr.smithi023.ehlpcl (mgr.14164) 219 : 2021-11-19T08:41:21.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:20 smithi023 conmon[33490]: cluster [DBG] pgmap v182: 65 pgs: 65 active+clean; 4.4 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 409 B/s wr, 2 op/s 2021-11-19T08:41:22.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:21 smithi134 conmon[36363]: audit 2021-11-19T08:41:20.916187+0000 mon.smithi023 (mon.0) 580 : audit 2021-11-19T08:41:22.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:21 smithi134 conmon[36363]: [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:41:22.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:21 smithi134 conmon[36363]: audit 2021-11-19T08:41:20.916581+0000 mon.smithi023 (mon.0) 581 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:41:22.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:21 smithi023 conmon[33490]: audit 2021-11-19T08:41:20.916187+0000 mon.smithi023 (mon.0) 2021-11-19T08:41:22.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:21 smithi023 conmon[33490]: 580 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:41:22.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:21 smithi023 conmon[33490]: audit 2021-11-19T08:41:20.916581+0000 mon.smithi023 (mon.0) 581 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:41:23.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:22 smithi134 conmon[36363]: cluster 2021-11-19T08:41:20.840882+0000 mgr.smithi023.ehlpcl (mgr.14164) 220 : cluster [DBG] pgmap v183: 65 pgs: 65 active+clean; 4.4 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 409 B/s wr, 2 op/s 2021-11-19T08:41:23.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:22 smithi023 conmon[33490]: cluster 2021-11-19T08:41:20.840882+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:41:23.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:22 smithi023 conmon[33490]: 220 : cluster [DBG] pgmap v183: 65 pgs: 65 active+clean; 4.4 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 409 B/s wr, 2 op/s 2021-11-19T08:41:25.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:24 smithi134 conmon[36363]: cluster 2021-11-19T08:41:22.841526+0000 mgr.smithi023.ehlpcl (mgr.14164) 221 : cluster [DBG] 2021-11-19T08:41:25.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:24 smithi134 conmon[36363]: pgmap v184: 65 pgs: 65 active+clean; 4.4 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 354 B/s wr, 1 op/s 2021-11-19T08:41:25.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:24 smithi023 conmon[33490]: cluster 2021-11-19T08:41:22.841526+0000 mgr.smithi023.ehlpcl (mgr.14164) 221 : cluster [DBG] pgmap v184: 65 pgs: 65 active+clean; 4.4 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 354 B/s wr, 1 op/s 2021-11-19T08:41:26.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:26.048+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:41:26.034 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:26.048+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-19T08:41:26.034 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:26.048+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-19T08:41:26.034 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:26.048+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-19T08:41:26.035 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:26.048+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-19T08:41:27.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:26 smithi134 conmon[36363]: cluster 2021-11-19T08:41:24.842071+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:41:27.065 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:26 smithi134 conmon[36363]: .14164) 222 : cluster [DBG] pgmap v185: 65 pgs: 65 active+clean; 4.4 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 341 B/s wr, 1 op/s 2021-11-19T08:41:27.065 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:26 smithi134 conmon[36363]: cluster 2021-11-19T08:41:26.307446+0000 mon.smithi023 (mon 2021-11-19T08:41:27.065 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:26 smithi134 conmon[36363]: .0) 582 : cluster [INF] daemon mds.cephfs.smithi134.jqsnzo restarted 2021-11-19T08:41:27.066 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:26 smithi134 conmon[36363]: cluster 2021-11 2021-11-19T08:41:27.066 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:26 smithi134 conmon[36363]: -19T08:41:26.314403 2021-11-19T08:41:27.066 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:26 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 2021-11-19T08:41:27.067 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:26 smithi134 conmon[36363]: 583 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2021-11-19T08:41:27.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:26 smithi023 conmon[33490]: cluster 2021-11-19T08:41:24.842071+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:41:27.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:26 smithi023 conmon[33490]: 14164) 222 : cluster [DBG] pgmap v185: 65 pgs: 65 active+clean; 4.4 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 341 B/s wr, 1 op/s 2021-11-19T08:41:27.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:26 smithi023 conmon[33490]: cluster 2021-11-19T08:41:26.307446+0000 mon.smithi023 (mon.0) 582 : cluster 2021-11-19T08:41:27.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:26 smithi023 conmon[33490]: [INF] daemon mds.cephfs.smithi134.jqsnzo restarted 2021-11-19T08:41:27.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:26 smithi023 conmon[33490]: cluster 2021- 2021-11-19T08:41:27.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:26 smithi023 conmon[33490]: 11-19T08:41:26.314403+0000 mon.smithi023 (mon.0) 583 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2021-11-19T08:41:27.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: cluster 2021-11-19T08:41:26.685510 2021-11-19T08:41:27.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 584 : cluster [DBG] mds.? [v2:172.21.15.134:6824/4111632986,v1:172.21.15.134:6825/4111632986] up:boot 2021-11-19T08:41:27.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: cluster 2021-11-19T08:41:26.685578+0000 2021-11-19T08:41:27.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: mon.smithi023 (mon.0) 585 : cluster [INF] Standby daemon mds.cephfs.smithi023.fuvzkr assigned to filesystem cephfs as rank 0 2021-11-19T08:41:27.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: cluster 2021-11-19T08:41:27.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: 2021-11-19T08:41:26.686017+0000 2021-11-19T08:41:27.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: mon.smithi023 (mon.0) 586 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi023.wazanl=up:active} 3 up:standby, 1 failed 2021-11-19T08:41:27.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: 2021-11-19T08:41:27.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: audit 2021-11-19T08:41:27.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: 2021-11-19T08:41:26.686137+0000 2021-11-19T08:41:27.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: mon.smithi023 (mon.0) 587 : audit [DBG] 2021-11-19T08:41:27.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.jqsnzo"}]: dispatch 2021-11-19T08:41:27.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: cluster 2021-11-19T08:41:26 2021-11-19T08:41:27.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: .690277+0000 mon.smithi023 (mon.0) 588 : 2021-11-19T08:41:27.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:27 smithi023 conmon[33490]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:replay,1=cephfs.smithi023.wazanl=up:active} 2 up:standby 2021-11-19T08:41:28.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:27 smithi134 conmon[36363]: cluster 2021-11-19T08:41:26.685510+0000 mon.smithi023 (mon.0) 584 : cluster [DBG] mds.? [v2:172.21.15.134:6824/4111632986,v1:172.21.15.134:6825/4111632986] up:boot 2021-11-19T08:41:28.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:27 smithi134 conmon[36363]: cluster 2021-11-19T08:41:26.685578+0000 2021-11-19T08:41:28.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:27 smithi134 conmon[36363]: mon.smithi023 (mon.0) 585 : cluster [INF] Standby daemon mds.cephfs.smithi023.fuvzkr assigned to filesystem cephfs as rank 0 2021-11-19T08:41:28.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:27 smithi134 conmon[36363]: cluster 2021-11-19T08:41:26.686017+0000 mon.smithi023 (mon.0) 586 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi023.wazanl=up:active} 3 up:standby, 1 failed 2021-11-19T08:41:28.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:27 smithi134 conmon[36363]: audit 2021-11-19T08:41 2021-11-19T08:41:28.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:27 smithi134 conmon[36363]: :26.686137+0000 mon.smithi023 (mon.0) 587 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.jqsnzo"}]: dispatch 2021-11-19T08:41:28.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:27 smithi134 conmon[36363]: cluster 2021-11 2021-11-19T08:41:28.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:27 smithi134 conmon[36363]: -19T08:41:26.690277+0000 mon.smithi023 ( 2021-11-19T08:41:28.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:27 smithi134 conmon[36363]: mon.0) 588 : cluster [DBG] 2021-11-19T08:41:28.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:27 smithi134 conmon[36363]: fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:replay,1=cephfs.smithi023.wazanl=up:active} 2 up:standby 2021-11-19T08:41:29.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:28 smithi134 conmon[36363]: cluster 2021-11-19T08:41:26.842592+0000 mgr.smithi023.ehlpcl (mgr.14164) 223 : cluster [DBG] 2021-11-19T08:41:29.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:28 smithi134 conmon[36363]: pgmap v187: 65 pgs: 65 active+clean; 4.4 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2021-11-19T08:41:29.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:28 smithi134 conmon[36363]: cluster 2021-11-19T08:41:27.694093+0000 mon.smithi023 (mon.0) 589 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3054538148,v1:172.21.15.23:6827/3054538148] up:resolve 2021-11-19T08:41:29.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:28 smithi134 conmon[36363]: cluster 2021-11-19T08:41:27.694166+0000 2021-11-19T08:41:29.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:28 smithi134 conmon[36363]: mon.smithi023 (mon.0) 590 : cluster [DBG] 2021-11-19T08:41:29.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:28 smithi134 conmon[36363]: fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi023.wazanl=up:active} 2 up:standby 2021-11-19T08:41:29.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:28 smithi023 conmon[33490]: cluster 2021-11-19T08:41:26.842592+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:41:29.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:28 smithi023 conmon[33490]: 223 : cluster [DBG] pgmap v187: 65 pgs: 65 active+clean; 4.4 KiB data, 32 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2021-11-19T08:41:29.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:28 smithi023 conmon[33490]: cluster 2021-11-19T08:41:27.694093+0000 mon.smithi023 (mon.0) 589 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3054538148,v1:172.21.15.23:6827/3054538148] up:resolve 2021-11-19T08:41:29.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:28 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:41:29.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:28 smithi023 conmon[33490]: 08:41:27.694166+0000 mon.smithi023 (mon.0) 590 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi023.wazanl=up:active} 2 up:standby 2021-11-19T08:41:31.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:30 smithi134 conmon[36363]: cluster 2021-11-19T08:41:28.843144+0000 mgr.smithi023.ehlpcl (mgr.14164) 224 : cluster [DBG] pgmap v188: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2021-11-19T08:41:31.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:30 smithi134 conmon[36363]: 2021-11-19T08:41:31.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:30 smithi023 conmon[33490]: cluster 2021-11-19 2021-11-19T08:41:31.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:30 smithi023 conmon[33490]: T08:41:28.843144+0000 mgr.smithi023.ehlpcl (mgr.14164) 224 : cluster [DBG] pgmap v188: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2021-11-19T08:41:32.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:31 smithi134 conmon[36363]: cluster 2021-11-19T08:41:30.709252+0000 mon.smithi023 (mon.0) 591 : cluster [DBG] mds.? [v2:172.21.15.134:6824/4111632986,v1:172.21.15.134:6825/4111632986] up:standby 2021-11-19T08:41:32.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:31 smithi134 conmon[36363]: cluster 2021-11-19T08:41:30.709339+0000 mon.smithi023 (mon.0) 592 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi023.wazanl=up:active} 2 up:standby 2021-11-19T08:41:32.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:31 smithi023 conmon[33490]: cluster 2021-11-19T08:41:30.709252+0000 mon.smithi023 (mon.0) 591 : cluster [DBG] mds.? [v2:172.21.15.134:6824/4111632986,v1:172.21.15.134:6825/4111632986] up:standby 2021-11-19T08:41:32.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:31 smithi023 conmon[33490]: cluster 2021-11-19T08:41:30.709339+0000 mon.smithi023 (mon.0) 592 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi023.wazanl=up:active} 2 up:standby 2021-11-19T08:41:33.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:32 smithi134 conmon[36363]: cluster 2021-11-19T08:41:30.843828+0000 mgr.smithi023.ehlpcl (mgr.14164) 225 : cluster [DBG] pgmap v189: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:41:33.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:32 smithi023 conmon[33490]: cluster 2021-11-19T08:41:30.843828+0000 mgr.smithi023.ehlpcl (mgr.14164) 225 : cluster [DBG] pgmap v189: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:41:35.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:34 smithi134 conmon[36363]: cluster 2021-11-19T08:41:32.844440+0000 mgr.smithi023.ehlpcl (mgr.14164) 226 : cluster [DBG] pgmap v190: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:41:35.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:34 smithi023 conmon[33490]: cluster 2021-11-19T08:41:32.844440+0000 mgr.smithi023.ehlpcl (mgr.14164) 226 : cluster [DBG] pgmap v190: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:41:37.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:36 smithi134 conmon[36363]: cluster 2021-11-19T08:41:34.845213+0000 mgr.smithi023.ehlpcl (mgr.14164) 227 : cluster [DBG] 2021-11-19T08:41:37.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:36 smithi134 conmon[36363]: pgmap v191: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:41:37.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:36 smithi023 conmon[33490]: cluster 2021-11-19T08:41:34.845213+0000 mgr.smithi023.ehlpcl (mgr.14164) 227 : cluster 2021-11-19T08:41:37.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:36 smithi023 conmon[33490]: [DBG] pgmap v191: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:41:39.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:38 smithi134 conmon[36363]: cluster 2021-11-19T08:41:36.845674+0000 mgr.smithi023.ehlpcl (mgr.14164) 228 : cluster [DBG] pgmap v192: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-19T08:41:39.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:38 smithi023 conmon[33490]: cluster 2021-11-19T08:41:36.845674+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:41:39.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:38 smithi023 conmon[33490]: 14164) 228 : cluster [DBG] pgmap v192: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-19T08:41:41.036 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:41.051+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:41:41.037 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:41.051+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-19T08:41:41.037 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:41.051+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-19T08:41:41.037 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:41.051+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-19T08:41:41.037 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:41.051+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-19T08:41:41.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:40 smithi134 conmon[36363]: cluster 2021-11-19T08:41:38.846102+0000 mgr.smithi023.ehlpcl (mgr.14164) 229 : cluster [DBG] pgmap v193: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:41:41.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:40 smithi023 conmon[33490]: cluster 2021-11-19T08:41:38.846102+0000 mgr.smithi023.ehlpcl (mgr.14164) 229 : cluster [DBG] pgmap v193: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:41:43.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:42 smithi134 conmon[36363]: cluster 2021-11-19T08:41:40.846854+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:41:43.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:42 smithi134 conmon[36363]: 230 : cluster [DBG] pgmap v194: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:41:43.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:42 smithi134 conmon[36363]: cluster 2021-11-19T08:41:42.422582+0000 mon.smithi023 (mon.0) 593 : cluster [INF] daemon mds.cephfs.smithi023.wazanl restarted 2021-11-19T08:41:43.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:42 smithi134 conmon[36363]: cluster 2021-11-19T08:41:42.431519+0000 mon.smithi023 (mon.0) 594 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2021-11-19T08:41:43.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:42 smithi134 conmon[36363]: audit 2021-11-19T08:41:42.505829+0000 mon.smithi023 (mon.0) 595 : audit 2021-11-19T08:41:43.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:42 smithi134 conmon[36363]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:41:43.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:42 smithi023 conmon[33490]: cluster 2021-11-19T08:41:40.846854+0000 2021-11-19T08:41:43.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:42 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 230 : cluster [DBG] pgmap v194: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:41:43.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:42 smithi023 conmon[33490]: cluster 2021-11-19T08:41:42.422582+0000 mon.smithi023 (mon.0) 593 : cluster [INF] daemon mds.cephfs.smithi023.wazanl restarted 2021-11-19T08:41:43.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:42 smithi023 conmon[33490]: cluster 2021-11-19T08:41:42.431519+0000 mon.smithi023 (mon.0) 594 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2021-11-19T08:41:43.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:42 smithi023 conmon[33490]: audit 2021-11-19T08:41 2021-11-19T08:41:43.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:42 smithi023 conmon[33490]: :42.505829+0000 mon.smithi023 (mon.0) 595 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:41:44.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:43 smithi134 conmon[36363]: cluster 2021-11-19T08:41:42.759160+0000 mon.smithi023 (mon.0) 596 : cluster [DBG] mds.? [v2:172.21.15.23:6828/747880251,v1:172.21.15.23:6829/747880251] up:boot 2021-11-19T08:41:44.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:43 smithi134 conmon[36363]: cluster 2021-11-19T08:41:42.759210+0000 mon.smithi023 (mon.0) 597 : cluster [INF] Standby daemon mds.cephfs.smithi134.opxllb assigned to filesystem cephfs as rank 1 2021-11-19T08:41:44.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:43 smithi134 conmon[36363]: cluster 2021-11-19T08:41:42.761722+0000 mon.smithi023 (mon.0) 598 : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi023.fuvzkr=up:resolve} 3 up:standby, 1 failed 2021-11-19T08:41:44.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:43 smithi134 conmon[36363]: audit 2021-11-19T08:41:42.761850+0000 mon.smithi023 (mon.0) 599 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.wazanl"}]: dispatch 2021-11-19T08:41:44.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:43 smithi134 conmon[36363]: cluster 2021-11-19T08:41:42.768659+0000 mon.smithi023 (mon.0) 600 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi134.opxllb=up:replay} 2 up:standby 2021-11-19T08:41:44.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:43 smithi134 conmon[36363]: audit 2021-11-19T08: 2021-11-19T08:41:44.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:43 smithi134 conmon[36363]: 41:42.906308+0000 mon.smithi023 (mon.0) 601 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:41:44.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:43 smithi023 conmon[33490]: cluster 2021-11-19T08:41:42.759160+0000 mon.smithi023 (mon.0) 596 : cluster [DBG] mds.? [v2:172.21.15.23:6828/747880251,v1:172.21.15.23:6829/747880251] up:boot 2021-11-19T08:41:44.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:43 smithi023 conmon[33490]: 2021-11-19T08:41:44.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:43 smithi023 conmon[33490]: cluster 2021-11-19T08:41:42.759210+0000 mon.smithi023 (mon.0) 597 : cluster [INF] Standby daemon mds.cephfs.smithi134.opxllb assigned to filesystem cephfs as rank 1 2021-11-19T08:41:44.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:43 smithi023 conmon[33490]: cluster 2021-11-19T08:41:42.761722+0000 mon.smithi023 (mon.0) 598 : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi023.fuvzkr=up:resolve} 3 up:standby, 1 failed 2021-11-19T08:41:44.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:43 smithi023 conmon[33490]: audit 2021-11-19T08:41:42.761850+0000 mon.smithi023 (mon.0) 599 : audit [DBG] 2021-11-19T08:41:44.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:43 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.wazanl"}]: dispatch 2021-11-19T08:41:44.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:43 smithi023 conmon[33490]: cluster 2021-11-19T08:41:44.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:43 smithi023 conmon[33490]: 2021-11-19T08: 2021-11-19T08:41:44.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:43 smithi023 conmon[33490]: 41:42.768659+0000 mon.smithi023 ( 2021-11-19T08:41:44.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:43 smithi023 conmon[33490]: mon.0) 600 : cluster [DBG] 2021-11-19T08:41:44.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:43 smithi023 conmon[33490]: fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi134.opxllb=up:replay} 2 up:standby 2021-11-19T08:41:44.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:43 smithi023 conmon[33490]: audit 2021-11-19T08: 2021-11-19T08:41:44.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:43 smithi023 conmon[33490]: 41:42.906308+0000 mon.smithi023 (mon.0) 601 : audit [INF] 2021-11-19T08:41:44.145 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:43 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:41:45.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:44 smithi134 conmon[36363]: cluster 2021-11-19T08:41:42.847174+0000 mgr.smithi023.ehlpcl (mgr.14164) 231 : cluster [DBG] pgmap v196: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:41:45.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:44 smithi134 conmon[36363]: cluster 2021-11-19T08:41:43.780841+0000 mon.smithi023 (mon.0) 602 : cluster [DBG] mds.? [v2:172.21.15.134:6826/954232334,v1:172.21.15.134:6827/954232334] up:resolve 2021-11-19T08:41:45.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:44 smithi134 conmon[36363]: cluster 2021-11-19T08:41:43.780901+0000 mon.smithi023 (mon.0) 603 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi134.opxllb=up:resolve} 2 up:standby 2021-11-19T08:41:45.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:44 smithi023 conmon[33490]: cluster 2021-11-19T08:41:42.847174+0000 mgr.smithi023.ehlpcl (mgr.14164) 231 : cluster 2021-11-19T08:41:45.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:44 smithi023 conmon[33490]: [DBG] pgmap v196: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:41:45.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:44 smithi023 conmon[33490]: cluster 2021-11-19T08:41:43.780841+0000 mon.smithi023 (mon.0) 602 : cluster [DBG] mds.? [v2:172.21.15.134:6826/954232334,v1:172.21.15.134:6827/954232334] up:resolve 2021-11-19T08:41:45.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:44 smithi023 conmon[33490]: cluster 2021-11-19T08:41:43.780901+0000 mon.smithi023 (mon.0) 603 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi134.opxllb=up:resolve} 2 up:standby 2021-11-19T08:41:46.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:45 smithi134 conmon[36363]: cluster 2021-11-19T08:41:44.782168+0000 mon.smithi023 (mon.0) 604 : cluster [DBG] 2021-11-19T08:41:46.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:45 smithi134 conmon[36363]: mds.? [v2:172.21.15.23:6826/3054538148,v1:172.21.15.23:6827/3054538148] up:reconnect 2021-11-19T08:41:46.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:45 smithi134 conmon[36363]: cluster 2021-11-19T08:41:44.782220+0000 mon.smithi023 (mon.0) 605 : cluster [DBG] mds.? [v2:172.21.15.134:6826/954232334,v1:172.21.15.134:6827/954232334] up:reconnect 2021-11-19T08:41:46.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:45 smithi134 conmon[36363]: cluster 2021-11-19T08:41:44.782289+0000 mon.smithi023 (mon.0) 606 : 2021-11-19T08:41:46.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:45 smithi134 conmon[36363]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:reconnect,1=cephfs.smithi134.opxllb=up:reconnect} 2 up:standby 2021-11-19T08:41:46.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:45 smithi134 conmon[36363]: cluster 2021-11-19T08:41:44.805369+0000 mds.cephfs.smithi023.fuvzkr (mds.0) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249998 2021-11-19T08:41:46.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:45 smithi134 conmon[36363]: cluster 2021-11-19T08:41:44 2021-11-19T08:41:46.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:45 smithi134 conmon[36363]: .805983+0000 mds.cephfs.smithi134.opxllb (mds.1) 1 2021-11-19T08:41:46.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:45 smithi134 conmon[36363]: : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249994 2021-11-19T08:41:46.113 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: cluster 2021-11-19T08:41:46.114 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: 2021-11-19T08:41:44.782168+0000 mon.smithi023 (mon.0) 604 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3054538148,v1:172.21.15.23:6827/3054538148] up:reconnect 2021-11-19T08:41:46.114 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: cluster 2021 2021-11-19T08:41:46.115 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: -11-19T08:41:44.782220+0000 mon.smithi023 (mon.0) 605 : cluster [DBG] mds.? [v2:172.21.15.134:6826/954232334,v1:172.21.15.134:6827/954232334] up:reconnect 2021-11-19T08:41:46.115 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: cluster 2021 2021-11-19T08:41:46.116 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: -11-19T08:41:44.782289+0000 mon.smithi023 (mon.0 2021-11-19T08:41:46.116 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: ) 606 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:reconnect,1=cephfs.smithi134.opxllb=up:reconnect} 2 up:standby 2021-11-19T08:41:46.117 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: cluster 2021-11- 2021-11-19T08:41:46.117 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: 19T08:41:44.805369+0000 mds.cephfs.smithi023.fuvzkr 2021-11-19T08:41:46.118 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: (mds.0) 1 : cluster [DBG] 2021-11-19T08:41:46.118 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249998 2021-11-19T08:41:46.118 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: cluster 2021- 2021-11-19T08:41:46.119 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: 11-19T08:41:44.805983 2021-11-19T08:41:46.119 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: +0000 mds.cephfs.smithi134.opxllb (mds.1) 1 : 2021-11-19T08:41:46.120 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:45 smithi023 conmon[33490]: cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249994 2021-11-19T08:41:47.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:46 smithi134 conmon[36363]: cluster 2021-11-19T08:41:44.847609+0000 mgr.smithi023.ehlpcl (mgr.14164) 232 : cluster 2021-11-19T08:41:47.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:46 smithi134 conmon[36363]: [DBG] pgmap v197: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-19T08:41:47.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:46 smithi134 conmon[36363]: cluster 2021-11-19T08:41:45.800290+0000 mon.smithi023 (mon.0) 607 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3054538148,v1:172.21.15.23:6827/3054538148] up:rejoin 2021-11-19T08:41:47.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:46 smithi134 conmon[36363]: cluster 2021-11-19T08:41:45.800354+0000 mon.smithi023 (mon.0) 608 : cluster [DBG] mds.? [v2:172.21.15.134:6826/954232334,v1:172.21.15.134:6827/954232334] up:rejoin 2021-11-19T08:41:47.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:46 smithi134 conmon[36363]: cluster 2021-11-19T08:41: 2021-11-19T08:41:47.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:46 smithi134 conmon[36363]: 45.800431+0000 mon.smithi023 (mon.0) 609 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:rejoin,1=cephfs.smithi134.opxllb=up:rejoin} 2 up:standby 2021-11-19T08:41:47.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:46 smithi134 conmon[36363]: cluster 2021-11-19T 2021-11-19T08:41:47.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:46 smithi134 conmon[36363]: 08:41:45.802184+0000 mon.smithi023 (mon.0) 610 : 2021-11-19T08:41:47.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:46 smithi134 conmon[36363]: cluster [INF] daemon mds.cephfs.smithi134.opxllb is now active in filesystem cephfs as rank 1 2021-11-19T08:41:47.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:46 smithi023 conmon[33490]: cluster 2021-11-19T08:41:44.847609+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:41:47.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:46 smithi023 conmon[33490]: 232 : cluster [DBG] pgmap v197: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-19T08:41:47.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:46 smithi023 conmon[33490]: cluster 2021-11-19T08:41:45.800290+0000 mon.smithi023 (mon.0) 607 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3054538148,v1:172.21.15.23:6827/3054538148] up:rejoin 2021-11-19T08:41:47.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:46 smithi023 conmon[33490]: cluster 2021-11-19T08:41:45.800354+0000 mon.smithi023 (mon.0) 608 : cluster [DBG] mds.? [v2:172.21.15.134:6826/954232334,v1:172.21.15.134:6827/954232334] up:rejoin 2021-11-19T08:41:47.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:46 smithi023 conmon[33490]: cluster 2021-11-19T08:41:45.800431+0000 mon.smithi023 (mon.0) 609 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:rejoin,1=cephfs.smithi134.opxllb=up:rejoin} 2 up:standby 2021-11-19T08:41:47.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:46 smithi023 conmon[33490]: cluster 2021-11-19T08:41:45.802184+0000 mon.smithi023 (mon.0) 610 : cluster [INF] daemon mds.cephfs.smithi134.opxllb is now active in filesystem cephfs as rank 1 2021-11-19T08:41:48.071 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:47 smithi023 conmon[33490]: cluster 2021-11-19T08:41:46.801062+0000 mon.smithi023 (mon 2021-11-19T08:41:48.071 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:47 smithi023 conmon[33490]: .0) 611 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3054538148,v1:172.21.15.23:6827/3054538148] up:clientreplay 2021-11-19T08:41:48.071 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:47 smithi023 conmon[33490]: cluster 2021-11-19T08:41:46.801117+0000 mon.smithi023 ( 2021-11-19T08:41:48.071 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:47 smithi023 conmon[33490]: mon.0) 612 : cluster [DBG] mds.? [v2:172.21.15.134:6826/954232334,v1:172.21.15.134:6827/954232334] up:active 2021-11-19T08:41:48.072 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:47 smithi023 conmon[33490]: cluster 2021-11-19 2021-11-19T08:41:48.072 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:47 smithi023 conmon[33490]: T08:41:46.801170+0000 mon.smithi023 ( 2021-11-19T08:41:48.072 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:47 smithi023 conmon[33490]: mon.0) 613 : cluster [DBG] 2021-11-19T08:41:48.073 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:47 smithi023 conmon[33490]: mds.? [v2:172.21.15.23:6828/747880251,v1:172.21.15.23:6829/747880251] up:standby 2021-11-19T08:41:48.073 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:47 smithi023 conmon[33490]: cluster 2021- 2021-11-19T08:41:48.073 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:47 smithi023 conmon[33490]: 11-19T08:41:46.801240 2021-11-19T08:41:48.073 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:47 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 614 2021-11-19T08:41:48.074 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:47 smithi023 conmon[33490]: : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:clientreplay,1=cephfs.smithi134.opxllb=up:active} 2 up:standby 2021-11-19T08:41:48.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:47 smithi134 conmon[36363]: cluster 2021-11-19T08:41:46.801062+0000 mon.smithi023 (mon.0) 611 : cluster [DBG] 2021-11-19T08:41:48.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:47 smithi134 conmon[36363]: mds.? [v2:172.21.15.23:6826/3054538148,v1:172.21.15.23:6827/3054538148] up:clientreplay 2021-11-19T08:41:48.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:47 smithi134 conmon[36363]: cluster 2021-11-19T08:41:46.801117+0000 mon.smithi023 (mon.0) 612 : cluster [DBG] mds.? [v2:172.21.15.134:6826/954232334,v1:172.21.15.134:6827/954232334] up:active 2021-11-19T08:41:48.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:47 smithi134 conmon[36363]: cluster 2021-11-19T08:41:48.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:47 smithi134 conmon[36363]: 2021-11-19T08:41:46.801170+0000 mon.smithi023 (mon.0) 613 : cluster 2021-11-19T08:41:48.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:47 smithi134 conmon[36363]: [DBG] mds.? [v2:172.21.15.23:6828/747880251,v1:172.21.15.23:6829/747880251] up:standby 2021-11-19T08:41:48.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:47 smithi134 conmon[36363]: cluster 2021-11-19T08:41:46 2021-11-19T08:41:48.291 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:47 smithi134 conmon[36363]: .801240+0000 mon.smithi023 (mon.0) 614 : cluster 2021-11-19T08:41:48.292 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:47 smithi134 conmon[36363]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:clientreplay,1=cephfs.smithi134.opxllb=up:active} 2 up:standby 2021-11-19T08:41:49.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:48 smithi023 conmon[33490]: cluster 2021-11-19T08:41:46.848098+0000 2021-11-19T08:41:49.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:48 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 233 : cluster [DBG] pgmap v198: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2021-11-19T08:41:49.289 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:48 smithi134 conmon[36363]: cluster 2021-11-19T08:41:46.848098+0000 mgr.smithi023.ehlpcl (mgr.14164) 233 : cluster [DBG] pgmap v198: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2021-11-19T08:41:51.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:50 smithi023 conmon[33490]: cluster 2021-11-19T08:41:48.848759+0000 mgr.smithi023.ehlpcl (mgr.14164) 234 2021-11-19T08:41:51.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:50 smithi023 conmon[33490]: : cluster [DBG] pgmap v199: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2021-11-19T08:41:51.289 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:50 smithi134 conmon[36363]: cluster 2021-11-19T08:41:48.848759+0000 mgr.smithi023.ehlpcl (mgr.14164) 234 : cluster [DBG] pgmap v199: 65 pgs: 65 active+clean; 4.4 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2021-11-19T08:41:52.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:51 smithi023 conmon[33490]: cluster 2021-11-19T08:41:50.849316+0000 mgr.smithi023.ehlpcl (mgr.14164) 235 : cluster [DBG] pgmap v200: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 409 B/s wr, 1 op/s 2021-11-19T08:41:52.289 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:51 smithi134 conmon[36363]: cluster 2021-11-19T08:41:50.849316+0000 mgr.smithi023.ehlpcl (mgr.14164) 235 : cluster [DBG] pgmap v200: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 409 B/s wr, 1 op/s 2021-11-19T08:41:54.289 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:53 smithi134 conmon[36363]: cluster 2021-11-19T08:41:52.849802+0000 mgr.smithi023.ehlpcl (mgr.14164) 236 : cluster [DBG] pgmap v201: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 393 B/s wr, 1 op/s 2021-11-19T08:41:54.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:53 smithi023 conmon[33490]: cluster 2021-11-19T08:41:52.849802+0000 mgr.smithi023.ehlpcl (mgr.14164) 236 : cluster 2021-11-19T08:41:54.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:53 smithi023 conmon[33490]: [DBG] pgmap v201: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 393 B/s wr, 1 op/s 2021-11-19T08:41:56.043 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:56.059+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:6826/530484646 2021-11-19T08:41:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:56.059+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-19T08:41:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:41:56.059+0000 7fc78bfff700 1 client.14556 ms_handle_refused on v2:172.21.15.23:6826/530484646 2021-11-19T08:41:56.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:56 smithi023 conmon[33490]: cluster 2021-11-19T08:41:54.850282+0000 mgr.smithi023.ehlpcl (mgr.14164) 237 : cluster [DBG] pgmap v202: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-19T08:41:56.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:56 smithi134 conmon[36363]: cluster 2021-11-19T08:41:54.850282+0000 mgr.smithi023.ehlpcl (mgr.14164) 237 : cluster [DBG] 2021-11-19T08:41:56.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:56 smithi134 conmon[36363]: pgmap v202: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-19T08:41:58.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:41:58 smithi134 conmon[36363]: cluster 2021-11-19T08:41:56.850866+0000 mgr.smithi023.ehlpcl (mgr.14164) 238 : cluster [DBG] pgmap v203: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 341 B/s wr, 0 op/s 2021-11-19T08:41:58.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:41:58 smithi023 conmon[33490]: cluster 2021-11-19T08:41:56.850866+0000 mgr.smithi023.ehlpcl (mgr.14164) 238 : cluster [DBG] pgmap v203: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 341 B/s wr, 0 op/s 2021-11-19T08:42:00.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:00 smithi134 conmon[36363]: cluster 2021-11-19T08:41:58.851289+0000 mgr.smithi023.ehlpcl (mgr.14164) 239 : cluster [DBG] pgmap v204: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-11-19T08:42:00.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:00 smithi134 conmon[36363]: 2021-11-19T08:42:00.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:00 smithi023 conmon[33490]: cluster 2021-11-19T08:41:58.851289+0000 mgr.smithi023.ehlpcl (mgr.14164) 239 : cluster [DBG] pgmap v204: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-11-19T08:42:02.477 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:02 smithi134 conmon[36363]: cluster 2021-11-19T08:42:00.851981+0000 mgr.smithi023.ehlpcl (mgr.14164) 240 : cluster [DBG] pgmap v205: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-11-19T08:42:02.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:02 smithi134 conmon[36363]: 2021-11-19T08:42:02.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:02 smithi134 conmon[36363]: cluster 2021-11-19T08:42:01.435603+0000 mon.smithi023 (mon.0) 615 : cluster [INF] daemon mds.cephfs.smithi023.fuvzkr restarted 2021-11-19T08:42:02.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:02 smithi134 conmon[36363]: cluster 2021-11-19T08:42:01.444389+0000 mon.smithi023 (mon.0) 616 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2021-11-19T08:42:02.561 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:02 smithi023 conmon[33490]: cluster 2021-11-19T08:42 2021-11-19T08:42:02.561 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:02 smithi023 conmon[33490]: :00.851981+0000 mgr.smithi023.ehlpcl (mgr.14164) 240 : cluster [DBG] pgmap v205: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-11-19T08:42:02.562 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:02 smithi023 conmon[33490]: cluster 2021-11-19T08:42:01.435603+0000 mon.smithi023 (mon.0) 615 : 2021-11-19T08:42:02.562 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:02 smithi023 conmon[33490]: cluster [INF] daemon mds.cephfs.smithi023.fuvzkr restarted 2021-11-19T08:42:02.562 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:02 smithi023 conmon[33490]: cluster 2021-11-19T08:42 2021-11-19T08:42:02.562 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:02 smithi023 conmon[33490]: :01.444389+0000 mon.smithi023 (mon.0) 616 : cluster [DBG] 2021-11-19T08:42:02.563 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:02 smithi023 conmon[33490]: osdmap e46: 6 total, 6 up, 6 in 2021-11-19T08:42:03.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:03 smithi023 conmon[33490]: cluster 2021-11-19T08:42:02.232778+0000 mon.smithi023 (mon.0) 617 : cluster 2021-11-19T08:42:03.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:03 smithi023 conmon[33490]: [DBG] mds.? [v2:172.21.15.23:6826/3365657753,v1:172.21.15.23:6827/3365657753] up:boot 2021-11-19T08:42:03.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:03 smithi023 conmon[33490]: cluster 2021-11-19T08:42:02.232838+0000 mon.smithi023 (mon.0) 618 : cluster [INF] Standby daemon mds.cephfs.smithi023.wazanl assigned to filesystem cephfs as rank 0 2021-11-19T08:42:03.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:03 smithi023 conmon[33490]: cluster 2021-11-19T08:42:02 2021-11-19T08:42:03.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:03 smithi023 conmon[33490]: .233409+0000 mon.smithi023 (mon.0) 619 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi134.opxllb=up:active} 3 up:standby, 1 failed 2021-11-19T08:42:03.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:03 smithi023 conmon[33490]: audit 2021-11-19T08:42:02.233560+0000 mon.smithi023 (mon.0) 620 : audit 2021-11-19T08:42:03.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:03 smithi023 conmon[33490]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.fuvzkr"}]: dispatch 2021-11-19T08:42:03.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:03 smithi023 conmon[33490]: cluster 2021-11-19T08:42:02.238082+0000 mon.smithi023 2021-11-19T08:42:03.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:03 smithi023 conmon[33490]: (mon.0) 621 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:replay,1=cephfs.smithi134.opxllb=up:active} 2 up:standby 2021-11-19T08:42:03.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:03 smithi134 conmon[36363]: cluster 2021-11-19T08:42:02.232778+0000 mon.smithi023 (mon.0) 617 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3365657753,v1:172.21.15.23:6827/3365657753] up:boot 2021-11-19T08:42:03.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:03 smithi134 conmon[36363]: cluster 2021-11-19T08:42:02.232838+0000 2021-11-19T08:42:03.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:03 smithi134 conmon[36363]: mon.smithi023 (mon.0) 618 : cluster [INF] Standby daemon mds.cephfs.smithi023.wazanl assigned to filesystem cephfs as rank 0 2021-11-19T08:42:03.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:03 smithi134 conmon[36363]: cluster 2021-11-19T08:42:02.233409+0000 mon.smithi023 (mon.0) 619 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi134.opxllb=up:active} 3 up:standby, 1 failed 2021-11-19T08:42:03.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:03 smithi134 conmon[36363]: audit 2021-11-19T08:42:02.233560+0000 mon.smithi023 (mon.0) 620 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.fuvzkr"}]: dispatch 2021-11-19T08:42:03.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:03 smithi134 conmon[36363]: cluster 2021-11-19T08:42:02.238082+0000 mon.smithi023 (mon.0) 621 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:replay,1=cephfs.smithi134.opxllb=up:active} 2 up:standby 2021-11-19T08:42:04.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:04 smithi134 conmon[36363]: cluster 2021-11-19T08:42:02.852563+0000 mgr.smithi023.ehlpcl (mgr.14164) 241 : cluster [DBG] 2021-11-19T08:42:04.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:04 smithi134 conmon[36363]: pgmap v207: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:42:04.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:04 smithi134 conmon[36363]: cluster 2021-11-19T08:42:03.244717+0000 mon.smithi023 (mon.0) 622 : cluster [DBG] mds.? [v2:172.21.15.23:6828/747880251,v1:172.21.15.23:6829/747880251] up:resolve 2021-11-19T08:42:04.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:04 smithi134 conmon[36363]: cluster 2021-11-19T08:42:03.244786+0000 mon.smithi023 (mon 2021-11-19T08:42:04.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:04 smithi134 conmon[36363]: .0) 623 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi134.opxllb=up:active} 2 up:standby 2021-11-19T08:42:04.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:04 smithi023 conmon[33490]: cluster 2021-11-19T08:42:02.852563+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:42:04.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:04 smithi023 conmon[33490]: mgr.14164) 241 : cluster [DBG] pgmap v207: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:42:04.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:04 smithi023 conmon[33490]: cluster 2021-11-19T08:42:03.244717+0000 mon.smithi023 (mon.0) 622 : 2021-11-19T08:42:04.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:04 smithi023 conmon[33490]: cluster [DBG] mds.? [v2:172.21.15.23:6828/747880251,v1:172.21.15.23:6829/747880251] up:resolve 2021-11-19T08:42:04.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:04 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:42:04.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:04 smithi023 conmon[33490]: 08:42:03.244786+0000 2021-11-19T08:42:04.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:04 smithi023 conmon[33490]: mon.smithi023 (mon.0) 623 : 2021-11-19T08:42:04.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:04 smithi023 conmon[33490]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi134.opxllb=up:active} 2 up:standby 2021-11-19T08:42:06.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:06 smithi023 conmon[33490]: cluster 2021-11-19T08:42:04.853275+0000 mgr.smithi023.ehlpcl (mgr.14164) 242 : cluster [DBG] 2021-11-19T08:42:06.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:06 smithi023 conmon[33490]: pgmap v208: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-19T08:42:06.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:06 smithi134 conmon[36363]: cluster 2021-11-19T08:42:04.853275+0000 mgr.smithi023.ehlpcl (mgr.14164) 242 : cluster [DBG] pgmap v208: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-19T08:42:07.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:07 smithi134 conmon[36363]: cluster 2021-11-19T08:42:06.253985+0000 mon.smithi023 (mon.0) 624 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3365657753,v1:172.21.15.23:6827/3365657753] up:standby 2021-11-19T08:42:07.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:07 smithi134 conmon[36363]: cluster 2021-11-19T08 2021-11-19T08:42:07.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:07 smithi134 conmon[36363]: :42:06.254090+0000 mon.smithi023 (mon.0) 625 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi134.opxllb=up:active} 2 up:standby 2021-11-19T08:42:07.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:07 smithi023 conmon[33490]: cluster 2021-11-19T08:42:06.253985+0000 mon.smithi023 (mon.0) 624 : 2021-11-19T08:42:07.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:07 smithi023 conmon[33490]: cluster [DBG] mds.? [v2:172.21.15.23:6826/3365657753,v1:172.21.15.23:6827/3365657753] up:standby 2021-11-19T08:42:07.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:07 smithi023 conmon[33490]: cluster 2021-11-19T08:42:06.254090+0000 mon.smithi023 (mon.0) 625 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi134.opxllb=up:active} 2 up:standby 2021-11-19T08:42:08.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:08 smithi134 conmon[36363]: cluster 2021-11-19T08:42:06.853733+0000 mgr.smithi023.ehlpcl (mgr.14164) 243 : cluster [DBG] pgmap v209: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:42:08.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:08 smithi023 conmon[33490]: cluster 2021-11-19T08:42:06.853733+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:42:08.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:08 smithi023 conmon[33490]: .14164) 243 : cluster [DBG] pgmap v209: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:42:10.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:10 smithi134 conmon[36363]: cluster 2021-11-19T08:42:08.854220+0000 mgr.smithi023.ehlpcl (mgr.14164) 244 : cluster [DBG] pgmap v210: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:42:10.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:10 smithi023 conmon[33490]: cluster 2021-11-19T08:42:08.854220+0000 mgr.smithi023.ehlpcl (mgr.14164) 244 : cluster [DBG] pgmap v210: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:42:11.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:11.062+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:11.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:11.062+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-19T08:42:11.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:11.062+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-19T08:42:11.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:11.062+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-19T08:42:11.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:11.062+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-19T08:42:12.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:12 smithi134 conmon[36363]: cluster 2021-11-19T08:42:10.854711+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:42:12.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:12 smithi134 conmon[36363]: .14164) 245 : cluster [DBG] pgmap v211: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:42:12.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:12 smithi023 conmon[33490]: cluster 2021-11-19T08:42:10.854711+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:42:12.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:12 smithi023 conmon[33490]: ) 245 : cluster [DBG] pgmap v211: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:42:14.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:14 smithi134 conmon[36363]: cluster 2021-11-19T08:42:12.855322+0000 mgr.smithi023.ehlpcl (mgr.14164) 246 : cluster [DBG] pgmap v212: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:14.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:14 smithi023 conmon[33490]: cluster 2021-11-19T08:42:12.855322+0000 mgr.smithi023.ehlpcl (mgr.14164) 246 : cluster [DBG] 2021-11-19T08:42:14.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:14 smithi023 conmon[33490]: pgmap v212: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:16.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:16 smithi023 conmon[33490]: cluster 2021-11-19T08:42:14.855959+0000 mgr.smithi023.ehlpcl (mgr.14164) 247 2021-11-19T08:42:16.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:16 smithi023 conmon[33490]: : cluster [DBG] pgmap v213: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:16.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:16 smithi134 conmon[36363]: cluster 2021-11-19T08:42:14.855959+0000 mgr.smithi023.ehlpcl (mgr.14164) 247 : cluster [DBG] pgmap v213: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:17.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:17 smithi023 conmon[33490]: cluster 2021-11-19T08:42:17.298968+0000 mon.smithi023 (mon.0) 626 : cluster 2021-11-19T08:42:17.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:17 smithi023 conmon[33490]: [INF] daemon mds.cephfs.smithi134.opxllb restarted 2021-11-19T08:42:17.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:17 smithi134 conmon[36363]: cluster 2021-11-19T08:42:17.298968+0000 mon.smithi023 (mon.0) 626 : cluster [INF] daemon mds.cephfs.smithi134.opxllb restarted 2021-11-19T08:42:17.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:17 smithi134 conmon[36363]: 2021-11-19T08:42:18.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: cluster 2021-11-19T08:42:16.856423+0000 mgr.smithi023.ehlpcl (mgr.14164) 248 : 2021-11-19T08:42:18.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: cluster [DBG] pgmap v214: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-19T08:42:18.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: cluster 2021-11-19T08:42:17.307382+0000 mon.smithi023 (mon.0) 627 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2021-11-19T08:42:18.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: cluster 2021-11-19T08:42:17.312077+0000 mon.smithi023 (mon. 2021-11-19T08:42:18.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: 0) 628 : cluster [DBG] mds.? [v2:172.21.15.134:6826/348816881,v1:172.21.15.134:6827/348816881] up:boot 2021-11-19T08:42:18.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: cluster 2021-11-19T08:42:18.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: 2021-11-19T08:42:17.312129+0000 mon.smithi023 (mon.0 2021-11-19T08:42:18.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: ) 629 : cluster 2021-11-19T08:42:18.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: [INF] Standby daemon mds.cephfs.smithi023.fuvzkr assigned to filesystem cephfs as rank 1 2021-11-19T08:42:18.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: cluster 2021-11-19T08:42:18.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: 2021-11-19T08:42 2021-11-19T08:42:18.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: :17.312399+0000 mon.smithi023 (mon.0) 630 2021-11-19T08:42:18.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi023.wazanl=up:resolve} 3 up:standby, 1 failed 2021-11-19T08:42:18.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: audit 2021-11-19T08:42:18.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: 2021-11-19T08:42 2021-11-19T08:42:18.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: :17.312651+0000 mon.smithi023 (mon.0) 631 : 2021-11-19T08:42:18.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.opxllb"}]: dispatch 2021-11-19T08:42:18.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: cluster 2021-11-19T08:42:18.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: 2021-11-19T08:42:17. 2021-11-19T08:42:18.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: 315516+0000 mon.smithi023 (mon.0) 632 : cluster 2021-11-19T08:42:18.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:18 smithi023 conmon[33490]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi023.fuvzkr=up:replay} 2 up:standby 2021-11-19T08:42:18.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:18 smithi134 conmon[36363]: cluster 2021-11-19T08:42:16.856423+0000 mgr.smithi023.ehlpcl (mgr.14164) 248 : cluster [DBG] pgmap v214: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-19T08:42:18.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:18 smithi134 conmon[36363]: cluster 2021-11-19T08:42:17.307382+0000 mon.smithi023 (mon.0) 627 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2021-11-19T08:42:18.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:18 smithi134 conmon[36363]: cluster 2021-11-19T08:42:17.312077+0000 mon.smithi023 (mon.0) 628 2021-11-19T08:42:18.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:18 smithi134 conmon[36363]: : cluster [DBG] mds.? [v2:172.21.15.134:6826/348816881,v1:172.21.15.134:6827/348816881] up:boot 2021-11-19T08:42:18.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:18 smithi134 conmon[36363]: cluster 2021-11-19T08:42:17.312129+0000 mon.smithi023 (mon.0 2021-11-19T08:42:18.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:18 smithi134 conmon[36363]: ) 629 : cluster [INF] Standby daemon mds.cephfs.smithi023.fuvzkr assigned to filesystem cephfs as rank 1 2021-11-19T08:42:18.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:18 smithi134 conmon[36363]: cluster 2021-11-19T08: 2021-11-19T08:42:18.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:18 smithi134 conmon[36363]: 42:17.312399+0000 mon.smithi023 (mon.0) 630 : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi023.wazanl=up:resolve} 3 up:standby, 1 failed 2021-11-19T08:42:18.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:18 smithi134 conmon[36363]: audit 2021-11-19T08:42:17.312651+0000 2021-11-19T08:42:18.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:18 smithi134 conmon[36363]: mon.smithi023 (mon.0) 631 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.opxllb"}]: dispatch 2021-11-19T08:42:18.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:18 smithi134 conmon[36363]: cluster 2021-11-19T08:42:17.315516+0000 mon.smithi023 2021-11-19T08:42:18.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:18 smithi134 conmon[36363]: (mon.0) 632 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi023.fuvzkr=up:replay} 2 up:standby 2021-11-19T08:42:19.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:19 smithi023 conmon[33490]: cluster 2021-11-19T08:42:18.323099+0000 mon.smithi023 (mon.0) 633 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3365657753,v1:172.21.15.23:6827/3365657753] up:resolve 2021-11-19T08:42:19.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:19 smithi023 conmon[33490]: cluster 2021-11-19T08:42:18.323156+0000 mon.smithi023 (mon.0) 634 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi023.fuvzkr=up:resolve} 2 up:standby 2021-11-19T08:42:19.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:19 smithi134 conmon[36363]: cluster 2021-11-19T08:42:18.323099+0000 mon.smithi023 (mon.0) 633 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3365657753,v1:172.21.15.23:6827/3365657753] up:resolve 2021-11-19T08:42:19.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:19 smithi134 conmon[36363]: cluster 2021-11-19T08:42:18.323156+0000 mon.smithi023 (mon.0) 634 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi023.fuvzkr=up:resolve} 2 up:standby 2021-11-19T08:42:20.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: cluster 2021-11-19T08:42:18.856902+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:42:20.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: .14164) 249 : cluster [DBG] pgmap v216: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:42:20.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: cluster 2021-11-19T08:42:19.331333+0000 mon.smithi023 (mon.0) 635 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3365657753,v1:172.21.15.23:6827/3365657753] up:reconnect 2021-11-19T08:42:20.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: 2021-11-19T08:42:20.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: cluster 2021-11-19T08:42:19 2021-11-19T08:42:20.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: .331400+0000 mon.smithi023 (mon.0) 636 : cluster [DBG] mds.? [v2:172.21.15.23:6828/747880251,v1:172.21.15.23:6829/747880251] up:reconnect 2021-11-19T08:42:20.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: 2021-11-19T08:42:20.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: cluster 2021 2021-11-19T08:42:20.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: -11-19T08:42:19.331472+0000 mon.smithi023 2021-11-19T08:42:20.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: (mon.0) 637 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:reconnect,1=cephfs.smithi023.fuvzkr=up:reconnect} 2 up:standby 2021-11-19T08:42:20.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: cluster 2021- 2021-11-19T08:42:20.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: 11-19T08:42:19.354854+0000 mds.cephfs.smithi023.wazanl 2021-11-19T08:42:20.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: (mds.0) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249998 2021-11-19T08:42:20.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: cluster 2021-11-19T08 2021-11-19T08:42:20.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: :42:19.354882+0000 mds.cephfs.smithi023.fuvzkr (mds.1) 1 2021-11-19T08:42:20.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:20 smithi023 conmon[33490]: : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249998 2021-11-19T08:42:20.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:20 smithi134 conmon[36363]: cluster 2021-11-19T08:42:18.856902+0000 mgr.smithi023.ehlpcl (mgr.14164) 249 : cluster [DBG] pgmap v216: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:42:20.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:20 smithi134 conmon[36363]: cluster 2021-11-19T08:42:19.331333+0000 mon.smithi023 (mon.0) 635 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3365657753,v1:172.21.15.23:6827/3365657753] up:reconnect 2021-11-19T08:42:20.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:20 smithi134 conmon[36363]: cluster 2021-11-19T08:42:19.331400+0000 2021-11-19T08:42:20.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:20 smithi134 conmon[36363]: mon.smithi023 (mon.0) 636 : cluster [DBG] mds.? [v2:172.21.15.23:6828/747880251,v1:172.21.15.23:6829/747880251] up:reconnect 2021-11-19T08:42:20.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:20 smithi134 conmon[36363]: cluster 2021-11-19T08:42: 2021-11-19T08:42:20.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:20 smithi134 conmon[36363]: 19.331472+0000 mon.smithi023 (mon.0) 637 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:reconnect,1=cephfs.smithi023.fuvzkr=up:reconnect} 2 up:standby 2021-11-19T08:42:20.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:20 smithi134 conmon[36363]: cluster 2021-11-19T08:42:19.354854+0000 mds.cephfs.smithi023.wazanl (mds.0) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249998 2021-11-19T08:42:20.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:20 smithi134 conmon[36363]: cluster 2021-11-19T08:42:19.354882+0000 mds.cephfs.smithi023.fuvzkr (mds.1) 1 : 2021-11-19T08:42:20.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:20 smithi134 conmon[36363]: cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249998 2021-11-19T08:42:21.543 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:21 smithi023 conmon[33490]: cluster 2021-11-19T08:42:20.336970+0000 mon.smithi023 (mon 2021-11-19T08:42:21.543 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:21 smithi023 conmon[33490]: .0) 638 : cluster [DBG] mds.? [v2:172.21.15.23:6828/747880251,v1:172.21.15.23:6829/747880251] up:rejoin 2021-11-19T08:42:21.543 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:21 smithi023 conmon[33490]: cluster 2021-11-19T08:42:20.337010+0000 mon.smithi023 (mon.0) 639 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3365657753,v1:172.21.15.23:6827/3365657753] up:rejoin 2021-11-19T08:42:21.543 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:21 smithi023 conmon[33490]: cluster 2021-11-19T08:42:20.337094+0000 mon.smithi023 (mon. 2021-11-19T08:42:21.544 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:21 smithi023 conmon[33490]: 0) 640 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:rejoin,1=cephfs.smithi023.fuvzkr=up:rejoin} 2 up:standby 2021-11-19T08:42:21.544 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:21 smithi023 conmon[33490]: audit 2021-11-19T08 2021-11-19T08:42:21.544 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:21 smithi023 conmon[33490]: :42:20.918056+0000 mon.smithi023 (mon.0 2021-11-19T08:42:21.544 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:21 smithi023 conmon[33490]: ) 641 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:42:21.545 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:21 smithi023 conmon[33490]: audit 2021-11- 2021-11-19T08:42:21.545 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:21 smithi023 conmon[33490]: 19T08:42:20.918500+0000 mon.smithi023 2021-11-19T08:42:21.545 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:21 smithi023 conmon[33490]: (mon.0) 642 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:42:21.545 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:21 smithi023 conmon[33490]: 2021-11-19T08:42:21.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:21 smithi134 conmon[36363]: cluster 2021-11-19T08:42:20.336970+0000 mon.smithi023 (mon.0) 638 : cluster [DBG] mds.? [v2:172.21.15.23:6828/747880251,v1:172.21.15.23:6829/747880251] up:rejoin 2021-11-19T08:42:21.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:21 smithi134 conmon[36363]: 2021-11-19T08:42:21.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:21 smithi134 conmon[36363]: cluster 2021-11-19T08:42:20.337010+0000 mon.smithi023 (mon.0) 639 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3365657753,v1:172.21.15.23:6827/3365657753] up:rejoin 2021-11-19T08:42:21.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:21 smithi134 conmon[36363]: cluster 2021-11-19T08:42 2021-11-19T08:42:21.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:21 smithi134 conmon[36363]: :20.337094+0000 mon.smithi023 (mon.0) 640 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:rejoin,1=cephfs.smithi023.fuvzkr=up:rejoin} 2 up:standby 2021-11-19T08:42:21.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:21 smithi134 conmon[36363]: audit 2021-11- 2021-11-19T08:42:21.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:21 smithi134 conmon[36363]: 19T08:42:20.918056+0000 mon.smithi023 (mon.0) 641 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:42:21.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:21 smithi134 conmon[36363]: 2021-11-19T08:42:21.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:21 smithi134 conmon[36363]: audit 2021-11-19T08:42:20.918500+0000 2021-11-19T08:42:21.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:21 smithi134 conmon[36363]: mon.smithi023 (mon.0) 642 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:42:22.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:22 smithi023 conmon[33490]: cluster 2021-11-19T08:42:20.857394+0000 mgr.smithi023.ehlpcl (mgr.14164) 250 : cluster [DBG] pgmap v217: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-19T08:42:22.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:22 smithi023 conmon[33490]: cluster 2021-11-19T08:42:21.346630+0000 mon.smithi023 (mon.0) 643 : cluster [DBG] mds.? [v2:172.21.15.134:6826/348816881,v1:172.21.15.134:6827/348816881] up:standby 2021-11-19T08:42:22.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:22 smithi023 conmon[33490]: cluster 2021-11-19T08:42:21.346701+0000 mon.smithi023 (mon.0) 2021-11-19T08:42:22.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:22 smithi023 conmon[33490]: 644 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:rejoin,1=cephfs.smithi023.fuvzkr=up:rejoin} 2 up:standby 2021-11-19T08:42:22.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:22 smithi134 conmon[36363]: cluster 2021-11-19T08:42:20.857394+0000 mgr.smithi023.ehlpcl (mgr.14164) 250 : cluster 2021-11-19T08:42:22.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:22 smithi134 conmon[36363]: [DBG] pgmap v217: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-19T08:42:22.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:22 smithi134 conmon[36363]: cluster 2021-11-19T08:42:21.346630+0000 mon.smithi023 (mon.0) 643 : cluster [DBG] mds.? [v2:172.21.15.134:6826/348816881,v1:172.21.15.134:6827/348816881] up:standby 2021-11-19T08:42:22.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:22 smithi134 conmon[36363]: cluster 2021-11-19T08:42:21 2021-11-19T08:42:22.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:22 smithi134 conmon[36363]: .346701+0000 mon.smithi023 (mon.0) 644 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:rejoin,1=cephfs.smithi023.fuvzkr=up:rejoin} 2 up:standby 2021-11-19T08:42:24.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:24 smithi023 conmon[33490]: cluster 2021-11-19T08:42:22.858090+0000 mgr.smithi023.ehlpcl (mgr.14164) 251 : cluster [DBG] pgmap v218: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:24.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:24 smithi134 conmon[36363]: cluster 2021-11-19T08:42:22.858090+0000 mgr.smithi023.ehlpcl (mgr.14164) 251 : cluster [DBG] pgmap v218: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:26.055 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:26.070+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:26.055 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:26.070+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-11-19T08:42:26.055 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:26.070+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-11-19T08:42:26.056 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:26.070+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-11-19T08:42:26.056 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:26.070+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-19T08:42:26.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:26 smithi023 conmon[33490]: cluster 2021-11-19T08:42:24.858590+0000 mgr.smithi023.ehlpcl (mgr.14164) 252 : cluster [DBG] pgmap v219: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:26.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:26 smithi134 conmon[36363]: cluster 2021-11-19T08:42:24.858590+0000 mgr.smithi023.ehlpcl (mgr.14164) 252 : cluster 2021-11-19T08:42:26.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:26 smithi134 conmon[36363]: [DBG] pgmap v219: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:28.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:28 smithi023 conmon[33490]: cluster 2021-11-19T08:42:26.859235+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:42:28.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:28 smithi023 conmon[33490]: 253 : cluster [DBG] pgmap v220: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:28.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:28 smithi134 conmon[36363]: cluster 2021-11-19T08:42:26.859235+0000 mgr.smithi023.ehlpcl (mgr.14164) 253 : cluster [DBG] 2021-11-19T08:42:28.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:28 smithi134 conmon[36363]: pgmap v220: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:30.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:30 smithi023 conmon[33490]: cluster 2021-11-19T08:42:28.859810+0000 mgr.smithi023.ehlpcl (mgr.14164) 254 : cluster [DBG] pgmap v221: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 974 B/s rd, 0 op/s 2021-11-19T08:42:30.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:30 smithi134 conmon[36363]: cluster 2021-11-19T08:42:28.859810+0000 mgr.smithi023.ehlpcl (mgr.14164) 254 : cluster [DBG] pgmap v221: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 974 B/s rd, 0 op/s 2021-11-19T08:42:32.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:32 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:42:32.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:32 smithi023 conmon[33490]: 08:42:30.860339+0000 mgr.smithi023.ehlpcl (mgr.14164) 255 : cluster [DBG] pgmap v222: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 0 op/s 2021-11-19T08:42:32.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:32 smithi023 conmon[33490]: cluster 2021-11-19T08:42: 2021-11-19T08:42:32.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:32 smithi023 conmon[33490]: 31.519207+0000 mon.smithi023 (mon.0) 645 : cluster [INF] daemon mds.cephfs.smithi023.wazanl restarted 2021-11-19T08:42:32.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:32 smithi023 conmon[33490]: cluster 2021-11-19T08 2021-11-19T08:42:32.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:32 smithi023 conmon[33490]: :42:31.525016+0000 mon.smithi023 (mon.0) 646 : cluster [DBG] 2021-11-19T08:42:32.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:32 smithi023 conmon[33490]: osdmap e48: 6 total, 6 up, 6 in 2021-11-19T08:42:32.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:32 smithi134 conmon[36363]: cluster 2021-11-19T08:42:30.860339+0000 mgr.smithi023.ehlpcl (mgr.14164) 255 : cluster [DBG] pgmap v222: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 0 op/s 2021-11-19T08:42:32.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:32 smithi134 conmon[36363]: cluster 2021-11-19T08:42:31.519207+0000 mon.smithi023 (mon.0) 645 : cluster [INF] daemon mds.cephfs.smithi023.wazanl restarted 2021-11-19T08:42:32.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:32 smithi134 conmon[36363]: cluster 2021-11-19T08:42:31.525016+0000 mon.smithi023 (mon.0) 646 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2021-11-19T08:42:33.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:33 smithi023 conmon[33490]: cluster 2021-11-19T08:42:32.379223+0000 mon.smithi023 (mon. 2021-11-19T08:42:33.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:33 smithi023 conmon[33490]: 0) 647 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3057422404,v1:172.21.15.23:6829/3057422404] up:boot 2021-11-19T08:42:33.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:33 smithi023 conmon[33490]: cluster 2021-11-19T08:42:32.379332+0000 mon.smithi023 (mon.0) 648 : cluster [INF] Standby daemon mds.cephfs.smithi134.jqsnzo assigned to filesystem cephfs as rank 0 2021-11-19T08:42:33.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:33 smithi023 conmon[33490]: cluster 2021-11-19 2021-11-19T08:42:33.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:33 smithi023 conmon[33490]: T08:42:32.379899+0000 mon.smithi023 (mon.0) 649 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi023.fuvzkr=up:rejoin} 3 up:standby, 1 failed 2021-11-19T08:42:33.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:33 smithi023 conmon[33490]: audit 2021-11-19T08:42:32.380055+0000 mon.smithi023 2021-11-19T08:42:33.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:33 smithi023 conmon[33490]: (mon.0) 650 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.wazanl"}]: dispatch 2021-11-19T08:42:33.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:33 smithi023 conmon[33490]: cluster 2021-11 2021-11-19T08:42:33.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:33 smithi023 conmon[33490]: -19T08:42:32.385374+0000 2021-11-19T08:42:33.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:33 smithi023 conmon[33490]: mon.smithi023 (mon.0) 651 : cluster [DBG] 2021-11-19T08:42:33.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:33 smithi023 conmon[33490]: fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:replay,1=cephfs.smithi023.fuvzkr=up:rejoin} 2 up:standby 2021-11-19T08:42:33.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:33 smithi134 conmon[36363]: cluster 2021-11-19T08:42:32.379223+0000 mon.smithi023 (mon.0) 647 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3057422404,v1:172.21.15.23:6829/3057422404] up:boot 2021-11-19T08:42:34.056 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:33 smithi134 conmon[36363]: cluster 2021-11-19T08:42:32.379332+0000 mon.smithi023 (mon.0) 648 : cluster [INF] Standby daemon mds.cephfs.smithi134.jqsnzo assigned to filesystem cephfs as rank 0 2021-11-19T08:42:34.056 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:33 smithi134 conmon[36363]: cluster 2021-11-19T08:42:32.379899+0000 mon.smithi023 (mon.0) 649 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi023.fuvzkr=up:rejoin} 3 up:standby, 1 failed 2021-11-19T08:42:34.057 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:33 smithi134 conmon[36363]: 2021-11-19T08:42:34.057 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:33 smithi134 conmon[36363]: audit 2021-11-19T08:42:32. 2021-11-19T08:42:34.057 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:33 smithi134 conmon[36363]: 380055+0000 mon.smithi023 (mon.0) 650 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.wazanl"}]: dispatch 2021-11-19T08:42:34.057 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:33 smithi134 conmon[36363]: cluster 2021- 2021-11-19T08:42:34.058 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:33 smithi134 conmon[36363]: 11-19T08:42:32.385374+0000 2021-11-19T08:42:34.058 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:33 smithi134 conmon[36363]: mon.smithi023 (mon.0) 651 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:replay,1=cephfs.smithi023.fuvzkr=up:rejoin} 2 up:standby 2021-11-19T08:42:34.536 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:34 smithi023 conmon[33490]: cluster 2021-11-19T08:42:32.860912+0000 mgr.smithi023.ehlpcl (mgr.14164) 256 2021-11-19T08:42:34.537 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:34 smithi023 conmon[33490]: : cluster [DBG] pgmap v224: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:42:34.537 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:34 smithi023 conmon[33490]: cluster 2021-11-19T08:42:33.390587+0000 mon.smithi023 (mon.0) 652 : cluster [DBG] mds.? [v2:172.21.15.134:6824/4111632986,v1:172.21.15.134:6825/4111632986] up:resolve 2021-11-19T08:42:34.538 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:34 smithi023 conmon[33490]: cluster 2021-11-19T08:42:33.390656+0000 mon.smithi023 (mon.0 2021-11-19T08:42:34.538 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:34 smithi023 conmon[33490]: ) 653 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.fuvzkr=up:rejoin} 2 up:standby 2021-11-19T08:42:34.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:34 smithi134 conmon[36363]: cluster 2021-11-19T08:42:32.860912+0000 mgr.smithi023.ehlpcl (mgr.14164) 256 : cluster [DBG] pgmap v224: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:42:34.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:34 smithi134 conmon[36363]: cluster 2021-11-19T08:42:33.390587+0000 mon.smithi023 (mon.0) 652 : cluster [DBG] mds.? [v2:172.21.15.134:6824/4111632986,v1:172.21.15.134:6825/4111632986] up:resolve 2021-11-19T08:42:34.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:34 smithi134 conmon[36363]: cluster 2021-11-19T08:42:33.390656+0000 mon.smithi023 (mon. 2021-11-19T08:42:34.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:34 smithi134 conmon[36363]: 0) 653 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.fuvzkr=up:rejoin} 2 up:standby 2021-11-19T08:42:36.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:36 smithi134 conmon[36363]: cluster 2021-11-19T08:42:34.861488+0000 mgr.smithi023.ehlpcl (mgr.14164) 257 : cluster [DBG] pgmap v225: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-19T08:42:36.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:36 smithi023 conmon[33490]: cluster 2021-11-19T08:42:34.861488+0000 mgr.smithi023.ehlpcl (mgr.14164) 257 : cluster [DBG] pgmap v225: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-19T08:42:37.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:37 smithi134 conmon[36363]: cluster 2021-11-19T08:42:36.403548+0000 mon.smithi023 (mon.0) 654 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3057422404,v1:172.21.15.23:6829/3057422404] up:standby 2021-11-19T08:42:37.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:37 smithi134 conmon[36363]: cluster 2021-11-19T08:42:36.403639+0000 mon.smithi023 (mon.0) 655 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.fuvzkr=up:rejoin} 2 up:standby 2021-11-19T08:42:37.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:37 smithi023 conmon[33490]: cluster 2021-11-19T08:42:36.403548+0000 mon.smithi023 (mon.0) 654 : cluster 2021-11-19T08:42:37.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:37 smithi023 conmon[33490]: [DBG] mds.? [v2:172.21.15.23:6828/3057422404,v1:172.21.15.23:6829/3057422404] up:standby 2021-11-19T08:42:37.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:37 smithi023 conmon[33490]: cluster 2021-11-19T08:42:36.403639+0000 mon.smithi023 (mon.0) 655 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.fuvzkr=up:rejoin} 2 up:standby 2021-11-19T08:42:38.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:38 smithi023 conmon[33490]: cluster 2021-11-19T08:42:36.861956+0000 mgr.smithi023.ehlpcl (mgr.14164) 258 : cluster 2021-11-19T08:42:38.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:38 smithi023 conmon[33490]: [DBG] pgmap v226: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:42:38.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:38 smithi134 conmon[36363]: cluster 2021-11-19T08:42:36.861956+0000 mgr.smithi023.ehlpcl (mgr.14164) 258 : cluster [DBG] pgmap v226: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:42:40.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:40 smithi134 conmon[36363]: cluster 2021-11-19T08:42:38.862570+0000 mgr.smithi023.ehlpcl (mgr.14164) 259 : cluster [DBG] pgmap v227: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:42:40.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:40 smithi023 conmon[33490]: cluster 2021-11-19T08:42:38.862570+0000 mgr.smithi023.ehlpcl (mgr.14164) 259 : 2021-11-19T08:42:40.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:40 smithi023 conmon[33490]: cluster [DBG] pgmap v227: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:42:41.057 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:41.073+0000 7fc7b108c700 1 -- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 msgr2=0x55a4c9e23890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:6826/530484646 2021-11-19T08:42:41.058 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:41.073+0000 7fc7b108c700 1 --2- 172.21.15.23:0/3267137691 >> [v2:172.21.15.23:6826/530484646,v1:172.21.15.23:6827/530484646] conn(0x55a4c9e214a0 0x55a4c9e23890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 15.000000 2021-11-19T08:42:41.058 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:41.073+0000 7fc78bfff700 1 client.14556 ms_handle_refused on v2:172.21.15.23:6826/530484646 2021-11-19T08:42:42.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:42 smithi023 conmon[33490]: cluster 2021-11-19T08:42:40.863169+0000 2021-11-19T08:42:42.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:42 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 260 : cluster [DBG] pgmap v228: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:42:42.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:42 smithi134 conmon[36363]: cluster 2021-11-19T08:42:40.863169+0000 mgr.smithi023.ehlpcl (mgr.14164) 260 : cluster [DBG] pgmap v228: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:42:43.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:43 smithi134 conmon[36363]: audit 2021-11-19T08:42:42.910512+0000 mon.smithi023 (mon.0) 656 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:42:43.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:43 smithi134 conmon[36363]: audit 2021-11-19T08:42:43.309350+0000 mon.smithi023 (mon.0) 657 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:42:43.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:43 smithi023 conmon[33490]: audit 2021-11-19T08:42:42.910512+0000 mon.smithi023 (mon.0) 656 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:42:43.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:43 smithi023 conmon[33490]: audit 2021-11-19T08:42:43.309350+0000 mon.smithi023 (mon.0) 657 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:42:44.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:44 smithi134 conmon[36363]: cluster 2021-11-19T08:42:42.863582+0000 mgr.smithi023.ehlpcl (mgr.14164) 261 : cluster [DBG] pgmap v229: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:44.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:44 smithi023 conmon[33490]: cluster 2021-11-19T08:42:42.863582+0000 mgr.smithi023.ehlpcl (mgr.14164) 261 : cluster [DBG] 2021-11-19T08:42:44.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:44 smithi023 conmon[33490]: pgmap v229: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:45.517 DEBUG:teuthology.orchestra.run:got remote process result: 124 2021-11-19T08:42:45.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:45 smithi134 conmon[36363]: cluster 2021-11-19T08:42:44.458980+0000 mon.smithi023 (mon.0) 658 : cluster [INF] daemon mds.cephfs.smithi023.fuvzkr restarted 2021-11-19T08:42:45.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:45 smithi134 conmon[36363]: cluster 2021-11-19T08:42:44.462734+0000 mon.smithi023 (mon.0) 659 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2021-11-19T08:42:45.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:45 smithi023 conmon[33490]: cluster 2021-11-19T08:42:44.458980+0000 mon.smithi023 (mon.0) 658 : cluster 2021-11-19T08:42:45.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:45 smithi023 conmon[33490]: [INF] daemon mds.cephfs.smithi023.fuvzkr restarted 2021-11-19T08:42:45.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:45 smithi023 conmon[33490]: cluster 2021-11-19T08:42:44.462734+0000 mon.smithi023 (mon.0) 659 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2021-11-19T08:42:46.519 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:42:46.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: cluster 2021-11-19T08:42:44.864060+0000 mgr.smithi023.ehlpcl (mgr.14164) 262 2021-11-19T08:42:46.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: : cluster [DBG] pgmap v231: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-19T08:42:46.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: cluster 2021-11-19T08:42:45.433996+0000 mon.smithi023 (mon.0) 660 : cluster [DBG] mds.? [v2:172.21.15.23:6826/2135889102,v1:172.21.15.23:6827/2135889102] up:boot 2021-11-19T08:42:46.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: cluster 2021-11-19T08:42:45.434052+0000 2021-11-19T08:42:46.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: mon.smithi023 (mon.0) 661 : cluster [INF] Standby daemon mds.cephfs.smithi023.wazanl assigned to filesystem cephfs as rank 1 2021-11-19T08:42:46.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: cluster 2021- 2021-11-19T08:42:46.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: 11-19T08:42:45.434344+0000 mon.smithi023 2021-11-19T08:42:46.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: (mon.0) 662 : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi134.jqsnzo=up:resolve} 3 up:standby, 1 failed 2021-11-19T08:42:46.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: audit 2021 2021-11-19T08:42:46.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: -11-19T08:42:45.434464+0000 2021-11-19T08:42:46.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: mon.smithi023 (mon.0) 663 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.fuvzkr"}]: dispatch 2021-11-19T08:42:46.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: cluster 2021 2021-11-19T08:42:46.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: -11-19T08:42:45.438054+0000 2021-11-19T08:42:46.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: mon.smithi023 (mon.0) 664 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.wazanl=up:replay} 2 up:standby 2021-11-19T08:42:46.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:46 smithi134 conmon[36363]: 2021-11-19T08:42:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.673+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1811885504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d80f5570 msgr2=0x7f82d80f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:46.661 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.673+0000 7f82dcc4e700 1 --2- 172.21.15.23:0/1811885504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d80f5570 0x7f82d80f5990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f82d0009a30 tx=0x7f82d0004c50).stop 2021-11-19T08:42:46.661 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.673+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1811885504 shutdown_connections 2021-11-19T08:42:46.662 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.673+0000 7f82dcc4e700 1 --2- 172.21.15.23:0/1811885504 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82d80f6810 0x7f82d80f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:46.662 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.673+0000 7f82dcc4e700 1 --2- 172.21.15.23:0/1811885504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d80f5570 0x7f82d80f5990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:46.662 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.673+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1811885504 >> 172.21.15.23:0/1811885504 conn(0x7f82d80f0cf0 msgr2=0x7f82d80f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:46.662 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.673+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1811885504 shutdown_connections 2021-11-19T08:42:46.663 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.673+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1811885504 wait complete. 2021-11-19T08:42:46.663 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82dcc4e700 1 Processor -- start 2021-11-19T08:42:46.663 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82dcc4e700 1 -- start start 2021-11-19T08:42:46.663 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82dcc4e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82d80f5570 0x7f82d80604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:46.664 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82dcc4e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d80f6810 0x7f82d810bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:46.664 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82dcc4e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d810c120 con 0x7f82d80f6810 2021-11-19T08:42:46.664 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82dcc4e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d810c260 con 0x7f82d80f5570 2021-11-19T08:42:46.664 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82d5d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d80f6810 0x7f82d810bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:46.665 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82d5d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d80f6810 0x7f82d810bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47514/0 (socket says 172.21.15.23:47514) 2021-11-19T08:42:46.665 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82d5d9b700 1 -- 172.21.15.23:0/1591699659 learned_addr learned my addr 172.21.15.23:0/1591699659 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:42:46.665 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82d659c700 1 --2- 172.21.15.23:0/1591699659 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82d80f5570 0x7f82d80604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:46.665 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82d5d9b700 1 -- 172.21.15.23:0/1591699659 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82d80f5570 msgr2=0x7f82d80604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:46.665 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82d5d9b700 1 --2- 172.21.15.23:0/1591699659 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82d80f5570 0x7f82d80604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82d5d9b700 1 -- 172.21.15.23:0/1591699659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c0005040 con 0x7f82d80f6810 2021-11-19T08:42:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82d659c700 1 --2- 172.21.15.23:0/1591699659 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82d80f5570 0x7f82d80604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:42:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82d5d9b700 1 --2- 172.21.15.23:0/1591699659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d80f6810 0x7f82d810bc30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f82c0004960 tx=0x7f82c0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.674+0000 7f82ceffd700 1 -- 172.21.15.23:0/1591699659 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82c0007e50 con 0x7f82d80f6810 2021-11-19T08:42:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.675+0000 7f82ceffd700 1 -- 172.21.15.23:0/1591699659 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82c000f040 con 0x7f82d80f6810 2021-11-19T08:42:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.675+0000 7f82ceffd700 1 -- 172.21.15.23:0/1591699659 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82c00149b0 con 0x7f82d80f6810 2021-11-19T08:42:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.675+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1591699659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d0009640 con 0x7f82d80f6810 2021-11-19T08:42:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.675+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1591699659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d810c8c0 con 0x7f82d80f6810 2021-11-19T08:42:46.668 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.675+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1591699659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d8027a00 con 0x7f82d80f6810 2021-11-19T08:42:46.668 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.676+0000 7f82ceffd700 1 -- 172.21.15.23:0/1591699659 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f82c0014be0 con 0x7f82d80f6810 2021-11-19T08:42:46.669 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.677+0000 7f82ceffd700 1 --2- 172.21.15.23:0/1591699659 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f82c405c2a0 0x7f82c405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:46.669 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.677+0000 7f82ceffd700 1 -- 172.21.15.23:0/1591699659 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f82c00767c0 con 0x7f82d80f6810 2021-11-19T08:42:46.669 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.677+0000 7f82d659c700 1 --2- 172.21.15.23:0/1591699659 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f82c405c2a0 0x7f82c405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:46.670 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.677+0000 7f82d659c700 1 --2- 172.21.15.23:0/1591699659 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f82c405c2a0 0x7f82c405e760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f82d000b3b0 tx=0x7f82d0004c50).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:46.670 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.680+0000 7f82ceffd700 1 -- 172.21.15.23:0/1591699659 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82c0049250 con 0x7f82d80f6810 2021-11-19T08:42:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.877+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1591699659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82d80fdd00 con 0x7f82d80f6810 2021-11-19T08:42:46.862 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.877+0000 7f82ceffd700 1 -- 172.21.15.23:0/1591699659 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2954 (secure 0 0 0) 0x7f82c0022ba0 con 0x7f82d80f6810 2021-11-19T08:42:46.863 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:42:46.863 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49},{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":51,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:42:46.437149+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24373,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:resolve","state_seq":5,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":46,"state":"up:resolve","state_seq":18,"addr":"172.21.15.134:6825/4111632986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":4111632986},{"type":"v1","addr":"172.21.15.134:6825","nonce":4111632986}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:42:46.864 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.879+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1591699659 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f82c405c2a0 msgr2=0x7f82c405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.879+0000 7f82dcc4e700 1 --2- 172.21.15.23:0/1591699659 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f82c405c2a0 0x7f82c405e760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f82d000b3b0 tx=0x7f82d0004c50).stop 2021-11-19T08:42:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.879+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1591699659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d80f6810 msgr2=0x7f82d810bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.879+0000 7f82dcc4e700 1 --2- 172.21.15.23:0/1591699659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d80f6810 0x7f82d810bc30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f82c0004960 tx=0x7f82c0009b30).stop 2021-11-19T08:42:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.879+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1591699659 shutdown_connections 2021-11-19T08:42:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.879+0000 7f82dcc4e700 1 --2- 172.21.15.23:0/1591699659 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82d80f5570 0x7f82d80604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.879+0000 7f82dcc4e700 1 --2- 172.21.15.23:0/1591699659 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f82c405c2a0 0x7f82c405e760 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.879+0000 7f82dcc4e700 1 --2- 172.21.15.23:0/1591699659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d80f6810 0x7f82d810bc30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.879+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1591699659 >> 172.21.15.23:0/1591699659 conn(0x7f82d80f0cf0 msgr2=0x7f82d80fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.879+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1591699659 shutdown_connections 2021-11-19T08:42:46.867 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:46.879+0000 7f82dcc4e700 1 -- 172.21.15.23:0/1591699659 wait complete. 2021-11-19T08:42:46.867 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 51 2021-11-19T08:42:46.874 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:46 smithi023 conmon[33490]: cluster 2021-11-19T08:42:44.864060+0000 mgr.smithi023.ehlpcl (mgr.14164) 262 : cluster [DBG] pgmap v231: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-19T08:42:46.875 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:46 smithi023 conmon[33490]: cluster 2021-11-19T08:42:45.433996+0000 mon.smithi023 (mon.0) 660 : cluster [DBG] mds.? [v2:172.21.15.23:6826/2135889102,v1:172.21.15.23:6827/2135889102] up:boot 2021-11-19T08:42:46.875 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:46 smithi023 conmon[33490]: cluster 2021-11-19T08:42:45.434052+0000 mon.smithi023 (mon.0) 661 : cluster [INF] Standby daemon mds.cephfs.smithi023.wazanl assigned to filesystem cephfs as rank 1 2021-11-19T08:42:46.875 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:46 smithi023 conmon[33490]: cluster 2021-11-19T08:42:45.434344+0000 2021-11-19T08:42:46.876 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:46 smithi023 conmon[33490]: mon.smithi023 (mon.0) 662 : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi134.jqsnzo=up:resolve} 3 up:standby, 1 failed 2021-11-19T08:42:46.876 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:46 smithi023 conmon[33490]: audit 2021-11-19T08:42:45.434464+0000 mon.smithi023 (mon.0) 663 : audit 2021-11-19T08:42:46.876 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:46 smithi023 conmon[33490]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.fuvzkr"}]: dispatch 2021-11-19T08:42:46.876 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:46 smithi023 conmon[33490]: cluster 2021-11-19T08:42:45.438054+0000 mon.smithi023 2021-11-19T08:42:46.876 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:46 smithi023 conmon[33490]: (mon.0) 664 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.wazanl=up:replay} 2 up:standby 2021-11-19T08:42:46.878 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:42:46.437149+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24373, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:resolve', 'state_seq': 5, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 46, 'state': 'up:resolve', 'state_seq': 18, 'addr': '172.21.15.134:6825/4111632986', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 4111632986}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 4111632986}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:42:46.878 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:resolve 2021-11-19T08:42:47.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:47 smithi134 conmon[36363]: cluster 2021-11-19T08:42:46.442233+0000 mon.smithi023 (mon.0) 665 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3057422404,v1:172.21.15.23:6829/3057422404] up:resolve 2021-11-19T08:42:47.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:47 smithi134 conmon[36363]: cluster 2021-11-19T08:42:46.442314+0000 mon.smithi023 2021-11-19T08:42:47.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:47 smithi134 conmon[36363]: (mon.0) 666 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.wazanl=up:resolve} 2 up:standby 2021-11-19T08:42:47.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:47 smithi134 conmon[36363]: audit 2021-11-19T08:42:46.878362+0000 mon.smithi023 (mon.0) 667 : audit [DBG] from='client.? 172.21.15.23:0/1591699659' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:47.881 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:42:47.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:47 smithi023 conmon[33490]: cluster 2021-11-19T08:42:46.442233+0000 2021-11-19T08:42:47.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:47 smithi023 conmon[33490]: mon.smithi023 (mon.0) 665 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3057422404,v1:172.21.15.23:6829/3057422404] up:resolve 2021-11-19T08:42:47.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:47 smithi023 conmon[33490]: cluster 2021-11-19T08:42:46.442314+0000 mon.smithi023 (mon.0) 666 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.wazanl=up:resolve} 2 up:standby 2021-11-19T08:42:47.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:47 smithi023 conmon[33490]: audit 2021-11-19T08:42:46.878362+0000 mon.smithi023 (mon.0) 667 : audit [DBG] from='client.? 172.21.15.23:0/1591699659' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:48.016 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.030+0000 7f98dccec700 1 -- 172.21.15.23:0/4008898371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98d80f6670 msgr2=0x7f98d80f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:48.017 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.030+0000 7f98dccec700 1 --2- 172.21.15.23:0/4008898371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98d80f6670 0x7f98d80f6a90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f98cc004660 tx=0x7f98cc009b30).stop 2021-11-19T08:42:48.017 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.030+0000 7f98dccec700 1 -- 172.21.15.23:0/4008898371 shutdown_connections 2021-11-19T08:42:48.017 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.030+0000 7f98dccec700 1 --2- 172.21.15.23:0/4008898371 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d80f7910 0x7f98d80f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:48.017 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.030+0000 7f98dccec700 1 --2- 172.21.15.23:0/4008898371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98d80f6670 0x7f98d80f6a90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:48.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.030+0000 7f98dccec700 1 -- 172.21.15.23:0/4008898371 >> 172.21.15.23:0/4008898371 conn(0x7f98d80f1db0 msgr2=0x7f98d80f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:48.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.030+0000 7f98dccec700 1 -- 172.21.15.23:0/4008898371 shutdown_connections 2021-11-19T08:42:48.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.030+0000 7f98dccec700 1 -- 172.21.15.23:0/4008898371 wait complete. 2021-11-19T08:42:48.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.030+0000 7f98dccec700 1 Processor -- start 2021-11-19T08:42:48.019 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.030+0000 7f98dccec700 1 -- start start 2021-11-19T08:42:48.019 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98dccec700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d80f6670 0x7f98d8106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:48.019 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98dccec700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98d80f7910 0x7f98d81073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:48.019 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98dccec700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d8107960 con 0x7f98d80f7910 2021-11-19T08:42:48.020 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98dccec700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d8107aa0 con 0x7f98d80f6670 2021-11-19T08:42:48.020 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98d5d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98d80f7910 0x7f98d81073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:48.020 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98d5d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98d80f7910 0x7f98d81073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47528/0 (socket says 172.21.15.23:47528) 2021-11-19T08:42:48.020 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98d5d9b700 1 -- 172.21.15.23:0/113272239 learned_addr learned my addr 172.21.15.23:0/113272239 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:42:48.020 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98d659c700 1 --2- 172.21.15.23:0/113272239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d80f6670 0x7f98d8106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:48.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98d5d9b700 1 -- 172.21.15.23:0/113272239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d80f6670 msgr2=0x7f98d8106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:48.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98d5d9b700 1 --2- 172.21.15.23:0/113272239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d80f6670 0x7f98d8106ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:48.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98d5d9b700 1 -- 172.21.15.23:0/113272239 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc005040 con 0x7f98d80f7910 2021-11-19T08:42:48.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98d5d9b700 1 --2- 172.21.15.23:0/113272239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98d80f7910 0x7f98d81073b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f98c80026d0 tx=0x7f98c8009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:48.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98c6ffd700 1 -- 172.21.15.23:0/113272239 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c80079c0 con 0x7f98d80f7910 2021-11-19T08:42:48.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98dccec700 1 -- 172.21.15.23:0/113272239 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d810c540 con 0x7f98d80f7910 2021-11-19T08:42:48.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98c6ffd700 1 -- 172.21.15.23:0/113272239 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98c800f040 con 0x7f98d80f7910 2021-11-19T08:42:48.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98c6ffd700 1 -- 172.21.15.23:0/113272239 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c80148b0 con 0x7f98d80f7910 2021-11-19T08:42:48.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.031+0000 7f98dccec700 1 -- 172.21.15.23:0/113272239 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d810cb00 con 0x7f98d80f7910 2021-11-19T08:42:48.023 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.032+0000 7f98dccec700 1 -- 172.21.15.23:0/113272239 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d8027a00 con 0x7f98d80f7910 2021-11-19T08:42:48.024 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.033+0000 7f98c6ffd700 1 -- 172.21.15.23:0/113272239 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f98c8007b20 con 0x7f98d80f7910 2021-11-19T08:42:48.024 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.033+0000 7f98c6ffd700 1 --2- 172.21.15.23:0/113272239 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f98c005c2a0 0x7f98c005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:48.024 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.033+0000 7f98c6ffd700 1 -- 172.21.15.23:0/113272239 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f98c8076160 con 0x7f98d80f7910 2021-11-19T08:42:48.025 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.033+0000 7f98d659c700 1 --2- 172.21.15.23:0/113272239 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f98c005c2a0 0x7f98c005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:48.025 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.034+0000 7f98d659c700 1 --2- 172.21.15.23:0/113272239 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f98c005c2a0 0x7f98c005e760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f98cc009f50 tx=0x7f98cc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:48.025 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.037+0000 7f98c6ffd700 1 -- 172.21.15.23:0/113272239 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98c8048b70 con 0x7f98d80f7910 2021-11-19T08:42:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.234+0000 7f98dccec700 1 -- 172.21.15.23:0/113272239 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98d80fee00 con 0x7f98d80f7910 2021-11-19T08:42:48.220 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.235+0000 7f98c6ffd700 1 -- 172.21.15.23:0/113272239 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2958 (secure 0 0 0) 0x7f98c804c190 con 0x7f98d80f7910 2021-11-19T08:42:48.220 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:42:48.221 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":49},{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":52,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:42:47.440605+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24373,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":46,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.134:6825/4111632986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":4111632986},{"type":"v1","addr":"172.21.15.134:6825","nonce":4111632986}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:42:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.236+0000 7f98dccec700 1 -- 172.21.15.23:0/113272239 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f98c005c2a0 msgr2=0x7f98c005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.236+0000 7f98dccec700 1 --2- 172.21.15.23:0/113272239 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f98c005c2a0 0x7f98c005e760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f98cc009f50 tx=0x7f98cc009b30).stop 2021-11-19T08:42:48.223 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.236+0000 7f98dccec700 1 -- 172.21.15.23:0/113272239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98d80f7910 msgr2=0x7f98d81073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:48.223 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.236+0000 7f98dccec700 1 --2- 172.21.15.23:0/113272239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98d80f7910 0x7f98d81073b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f98c80026d0 tx=0x7f98c8009de0).stop 2021-11-19T08:42:48.223 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.236+0000 7f98dccec700 1 -- 172.21.15.23:0/113272239 shutdown_connections 2021-11-19T08:42:48.223 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.236+0000 7f98dccec700 1 --2- 172.21.15.23:0/113272239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98d80f6670 0x7f98d8106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:48.223 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.236+0000 7f98dccec700 1 --2- 172.21.15.23:0/113272239 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f98c005c2a0 0x7f98c005e760 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:48.224 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.236+0000 7f98dccec700 1 --2- 172.21.15.23:0/113272239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98d80f7910 0x7f98d81073b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:48.224 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.236+0000 7f98dccec700 1 -- 172.21.15.23:0/113272239 >> 172.21.15.23:0/113272239 conn(0x7f98d80f1db0 msgr2=0x7f98d80fb360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:48.224 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.237+0000 7f98dccec700 1 -- 172.21.15.23:0/113272239 shutdown_connections 2021-11-19T08:42:48.224 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:48.237+0000 7f98dccec700 1 -- 172.21.15.23:0/113272239 wait complete. 2021-11-19T08:42:48.225 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 52 2021-11-19T08:42:48.233 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:42:47.440605+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24373, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:reconnect', 'state_seq': 6, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 46, 'state': 'up:reconnect', 'state_seq': 22, 'addr': '172.21.15.134:6825/4111632986', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 4111632986}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 4111632986}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:42:48.234 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:reconnect 2021-11-19T08:42:48.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:48 smithi134 conmon[36363]: cluster 2021-11-19T08:42:46.864499+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:42:48.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:48 smithi134 conmon[36363]: 263 : cluster [DBG] pgmap v232: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-19T08:42:48.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:48 smithi134 conmon[36363]: cluster 2021-11-19T08:42:47.448350+0000 mon.smithi023 (mon.0) 668 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3057422404,v1:172.21.15.23:6829/3057422404] up:reconnect 2021-11-19T08:42:48.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:48 smithi134 conmon[36363]: cluster 2021-11-19T08:42:47. 2021-11-19T08:42:48.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:48 smithi134 conmon[36363]: 448405+0000 mon.smithi023 (mon.0) 669 : cluster [DBG] mds.? [v2:172.21.15.134:6824/4111632986,v1:172.21.15.134:6825/4111632986] up:reconnect 2021-11-19T08:42:48.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:48 smithi134 conmon[36363]: cluster 2021-11-19T08:42:47.448491+0000 mon.smithi023 ( 2021-11-19T08:42:48.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:48 smithi134 conmon[36363]: mon.0) 670 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:reconnect,1=cephfs.smithi023.wazanl=up:reconnect} 2 up:standby 2021-11-19T08:42:48.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:48 smithi134 conmon[36363]: cluster 2021 2021-11-19T08:42:48.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:48 smithi134 conmon[36363]: -11-19T08:42:47.471090+0000 mds.cephfs.smithi023.wazanl (mds.1) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249998 2021-11-19T08:42:48.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:48 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:42:48.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:48 smithi134 conmon[36363]: 11-19T08:42:48.235977+0000 mon.smithi023 (mon.0) 671 : 2021-11-19T08:42:48.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:48 smithi134 conmon[36363]: audit [DBG] from='client.? 172.21.15.23:0/113272239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:48.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: cluster 2021-11-19T08:42:46.864499+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:42:48.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: ) 263 : cluster [DBG] pgmap v232: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-19T08:42:48.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: cluster 2021-11-19T08:42:47.448350+0000 mon.smithi023 (mon.0) 668 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3057422404,v1:172.21.15.23:6829/3057422404] up:reconnect 2021-11-19T08:42:48.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:42:48.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: 08:42:47.448405+0000 mon.smithi023 (mon.0) 669 : cluster [DBG] mds.? [v2:172.21.15.134:6824/4111632986,v1:172.21.15.134:6825/4111632986] up:reconnect 2021-11-19T08:42:48.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: cluster 2021-11 2021-11-19T08:42:48.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: -19T08:42:47.448491+0000 mon.smithi023 (mon.0) 670 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:reconnect,1=cephfs.smithi023.wazanl=up:reconnect} 2 up:standby 2021-11-19T08:42:48.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: 2021-11-19T08:42:48.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: cluster 2021-11-19T08:42:48.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: 2021-11-19T08:42:47. 2021-11-19T08:42:48.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: 471090+0000 mds.cephfs.smithi023.wazanl (mds.1) 1 : cluster [DBG] 2021-11-19T08:42:48.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249998 2021-11-19T08:42:48.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: audit 2021-11-19T08:42:48 2021-11-19T08:42:48.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: .235977+0000 mon.smithi023 (mon.0) 671 : audit [DBG] from='client.? 172.21.15.23:0/113272239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:48.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:48 smithi023 conmon[33490]: 2021-11-19T08:42:49.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:42:49.374 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.387+0000 7f12038df700 1 -- 172.21.15.23:0/2754492679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11fc0f5580 msgr2=0x7f11fc0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:49.374 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.387+0000 7f12038df700 1 --2- 172.21.15.23:0/2754492679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11fc0f5580 0x7f11fc0f59a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f11f0004660 tx=0x7f11f0009b30).stop 2021-11-19T08:42:49.374 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.387+0000 7f12038df700 1 -- 172.21.15.23:0/2754492679 shutdown_connections 2021-11-19T08:42:49.375 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.387+0000 7f12038df700 1 --2- 172.21.15.23:0/2754492679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f11fc0f6820 0x7f11fc0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:49.375 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.387+0000 7f12038df700 1 --2- 172.21.15.23:0/2754492679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11fc0f5580 0x7f11fc0f59a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:49.375 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.387+0000 7f12038df700 1 -- 172.21.15.23:0/2754492679 >> 172.21.15.23:0/2754492679 conn(0x7f11fc0f0ce0 msgr2=0x7f11fc0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:49.375 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.387+0000 7f12038df700 1 -- 172.21.15.23:0/2754492679 shutdown_connections 2021-11-19T08:42:49.376 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.387+0000 7f12038df700 1 -- 172.21.15.23:0/2754492679 wait complete. 2021-11-19T08:42:49.376 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.388+0000 7f12038df700 1 Processor -- start 2021-11-19T08:42:49.376 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.388+0000 7f12038df700 1 -- start start 2021-11-19T08:42:49.376 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.388+0000 7f12038df700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f11fc0f5580 0x7f11fc105db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:49.377 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.388+0000 7f12038df700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11fc0f6820 0x7f11fc1062a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:49.377 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.388+0000 7f12038df700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc106850 con 0x7f11fc0f6820 2021-11-19T08:42:49.377 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.388+0000 7f12038df700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc106990 con 0x7f11fc0f5580 2021-11-19T08:42:49.377 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.388+0000 7f1200e7a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11fc0f6820 0x7f11fc1062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:49.378 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.388+0000 7f1200e7a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11fc0f6820 0x7f11fc1062a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47546/0 (socket says 172.21.15.23:47546) 2021-11-19T08:42:49.378 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.388+0000 7f1200e7a700 1 -- 172.21.15.23:0/3529608456 learned_addr learned my addr 172.21.15.23:0/3529608456 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:42:49.378 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.388+0000 7f120167b700 1 --2- 172.21.15.23:0/3529608456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f11fc0f5580 0x7f11fc105db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:49.378 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.388+0000 7f1200e7a700 1 -- 172.21.15.23:0/3529608456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f11fc0f5580 msgr2=0x7f11fc105db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:49.379 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.388+0000 7f1200e7a700 1 --2- 172.21.15.23:0/3529608456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f11fc0f5580 0x7f11fc105db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:49.379 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.389+0000 7f1200e7a700 1 -- 172.21.15.23:0/3529608456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f0005040 con 0x7f11fc0f6820 2021-11-19T08:42:49.379 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.389+0000 7f120167b700 1 --2- 172.21.15.23:0/3529608456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f11fc0f5580 0x7f11fc105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:42:49.379 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.389+0000 7f1200e7a700 1 --2- 172.21.15.23:0/3529608456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11fc0f6820 0x7f11fc1062a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f11f800c530 tx=0x7f11f8007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:49.380 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.389+0000 7f11edffb700 1 -- 172.21.15.23:0/3529608456 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11f800f650 con 0x7f11fc0f6820 2021-11-19T08:42:49.380 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.389+0000 7f11edffb700 1 -- 172.21.15.23:0/3529608456 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11f800fc60 con 0x7f11fc0f6820 2021-11-19T08:42:49.380 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.389+0000 7f12038df700 1 -- 172.21.15.23:0/3529608456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11fc10b430 con 0x7f11fc0f6820 2021-11-19T08:42:49.380 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.389+0000 7f11edffb700 1 -- 172.21.15.23:0/3529608456 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11f8012d30 con 0x7f11fc0f6820 2021-11-19T08:42:49.380 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.389+0000 7f12038df700 1 -- 172.21.15.23:0/3529608456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc10b9f0 con 0x7f11fc0f6820 2021-11-19T08:42:49.382 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.390+0000 7f12038df700 1 -- 172.21.15.23:0/3529608456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11fc027a00 con 0x7f11fc0f6820 2021-11-19T08:42:49.383 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.390+0000 7f11edffb700 1 -- 172.21.15.23:0/3529608456 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f11f8010070 con 0x7f11fc0f6820 2021-11-19T08:42:49.383 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.391+0000 7f11edffb700 1 --2- 172.21.15.23:0/3529608456 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f11e805c2a0 0x7f11e805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:49.383 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.391+0000 7f11edffb700 1 -- 172.21.15.23:0/3529608456 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f11f8075f00 con 0x7f11fc0f6820 2021-11-19T08:42:49.383 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.391+0000 7f120167b700 1 --2- 172.21.15.23:0/3529608456 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f11e805c2a0 0x7f11e805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:49.384 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.391+0000 7f120167b700 1 --2- 172.21.15.23:0/3529608456 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f11e805c2a0 0x7f11e805e760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f11f0009d60 tx=0x7f11f0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:49.384 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.395+0000 7f11edffb700 1 -- 172.21.15.23:0/3529608456 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11f80452b0 con 0x7f11fc0f6820 2021-11-19T08:42:49.528 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:49 smithi023 conmon[33490]: cluster 2021-11-19T08:42:48.449383 2021-11-19T08:42:49.529 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:49 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 672 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3057422404,v1:172.21.15.23:6829/3057422404] up:rejoin 2021-11-19T08:42:49.529 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:49 smithi023 conmon[33490]: cluster 2021-11-19T08:42: 2021-11-19T08:42:49.529 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:49 smithi023 conmon[33490]: 48.449433+0000 mon.smithi023 (mon.0) 673 : cluster [DBG] mds.? [v2:172.21.15.134:6824/4111632986,v1:172.21.15.134:6825/4111632986] up:rejoin 2021-11-19T08:42:49.529 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:49 smithi023 conmon[33490]: cluster 2021-11-19T08:42:48.449511 2021-11-19T08:42:49.530 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:49 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 674 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:rejoin,1=cephfs.smithi023.wazanl=up:rejoin} 2 up:standby 2021-11-19T08:42:49.595 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.610+0000 7f12038df700 1 -- 172.21.15.23:0/3529608456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11fc0fdd10 con 0x7f11fc0f6820 2021-11-19T08:42:49.595 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.611+0000 7f11edffb700 1 -- 172.21.15.23:0/3529608456 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f11f8048910 con 0x7f11fc0f6820 2021-11-19T08:42:49.596 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:42:49.596 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":53,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:42:48.445915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24373,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":46,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.134:6825/4111632986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":4111632986},{"type":"v1","addr":"172.21.15.134:6825","nonce":4111632986}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:42:49.598 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.612+0000 7f12038df700 1 -- 172.21.15.23:0/3529608456 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f11e805c2a0 msgr2=0x7f11e805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:49.598 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.612+0000 7f12038df700 1 --2- 172.21.15.23:0/3529608456 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f11e805c2a0 0x7f11e805e760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f11f0009d60 tx=0x7f11f0009b30).stop 2021-11-19T08:42:49.598 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.612+0000 7f12038df700 1 -- 172.21.15.23:0/3529608456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11fc0f6820 msgr2=0x7f11fc1062a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:49.598 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.612+0000 7f12038df700 1 --2- 172.21.15.23:0/3529608456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11fc0f6820 0x7f11fc1062a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f11f800c530 tx=0x7f11f8007e20).stop 2021-11-19T08:42:49.599 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.613+0000 7f12038df700 1 -- 172.21.15.23:0/3529608456 shutdown_connections 2021-11-19T08:42:49.599 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.613+0000 7f12038df700 1 --2- 172.21.15.23:0/3529608456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f11fc0f5580 0x7f11fc105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:49.599 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.613+0000 7f12038df700 1 --2- 172.21.15.23:0/3529608456 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f11e805c2a0 0x7f11e805e760 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:49.599 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.613+0000 7f12038df700 1 --2- 172.21.15.23:0/3529608456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11fc0f6820 0x7f11fc1062a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:49.599 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.613+0000 7f12038df700 1 -- 172.21.15.23:0/3529608456 >> 172.21.15.23:0/3529608456 conn(0x7f11fc0f0ce0 msgr2=0x7f11fc0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:49.600 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.613+0000 7f12038df700 1 -- 172.21.15.23:0/3529608456 shutdown_connections 2021-11-19T08:42:49.600 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:49.613+0000 7f12038df700 1 -- 172.21.15.23:0/3529608456 wait complete. 2021-11-19T08:42:49.600 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 54 2021-11-19T08:42:49.609 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:42:48.445915+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24373, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 46, 'state': 'up:rejoin', 'state_seq': 23, 'addr': '172.21.15.134:6825/4111632986', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 4111632986}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 4111632986}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:42:49.610 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:42:49.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:49 smithi134 conmon[36363]: cluster 2021-11-19T08:42:48.449383+0000 mon.smithi023 (mon.0) 672 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3057422404,v1:172.21.15.23:6829/3057422404] up:rejoin 2021-11-19T08:42:49.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:49 smithi134 conmon[36363]: cluster 2021-11-19T08:42:48.449433+0000 mon.smithi023 (mon.0) 673 : cluster [DBG] mds.? [v2:172.21.15.134:6824/4111632986,v1:172.21.15.134:6825/4111632986] up:rejoin 2021-11-19T08:42:49.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:49 smithi134 conmon[36363]: cluster 2021-11-19T08:42:48.449511+0000 mon.smithi023 (mon.0) 674 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:rejoin,1=cephfs.smithi023.wazanl=up:rejoin} 2 up:standby 2021-11-19T08:42:50.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:42:50.746 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.759+0000 7f5f90f4e700 1 -- 172.21.15.23:0/463601930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f8c0f4780 msgr2=0x7f5f8c0f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.759+0000 7f5f90f4e700 1 --2- 172.21.15.23:0/463601930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f8c0f4780 0x7f5f8c0f4c00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f5f74004660 tx=0x7f5f74009b30).stop 2021-11-19T08:42:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.759+0000 7f5f90f4e700 1 -- 172.21.15.23:0/463601930 shutdown_connections 2021-11-19T08:42:50.748 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.759+0000 7f5f90f4e700 1 --2- 172.21.15.23:0/463601930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f8c0f4780 0x7f5f8c0f4c00 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:50.748 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.759+0000 7f5f90f4e700 1 --2- 172.21.15.23:0/463601930 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f8c0f7d90 0x7f5f8c0f81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-19T08:42:50.748 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.759+0000 7f5f90f4e700 1 -- 172.21.15.23:0/463601930 >> 172.21.15.23:0/463601930 conn(0x7f5f8c0f04a0 msgr2=0x7f5f8c0f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:50.748 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.759+0000 7f5f90f4e700 1 -- 172.21.15.23:0/463601930 shutdown_connections 2021-11-19T08:42:50.749 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.759+0000 7f5f90f4e700 1 -- 172.21.15.23:0/463601930 wait complete. 2021-11-19T08:42:50.749 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.760+0000 7f5f90f4e700 1 Processor -- start 2021-11-19T08:42:50.749 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.760+0000 7f5f90f4e700 1 -- start start 2021-11-19T08:42:50.749 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.760+0000 7f5f90f4e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f8c0f4780 0x7f5f8c0f7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:50.750 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.760+0000 7f5f90f4e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f8c0f7d90 0x7f5f8c0f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:50.750 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.760+0000 7f5f90f4e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0f5ea0 con 0x7f5f8c0f7d90 2021-11-19T08:42:50.750 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.760+0000 7f5f90f4e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0f5fe0 con 0x7f5f8c0f4780 2021-11-19T08:42:50.750 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.760+0000 7f5f89d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f8c0f7d90 0x7f5f8c0f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:50.750 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.760+0000 7f5f89d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f8c0f7d90 0x7f5f8c0f59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47562/0 (socket says 172.21.15.23:47562) 2021-11-19T08:42:50.751 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.760+0000 7f5f89d9b700 1 -- 172.21.15.23:0/4005100079 learned_addr learned my addr 172.21.15.23:0/4005100079 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:42:50.751 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.760+0000 7f5f8a59c700 1 --2- 172.21.15.23:0/4005100079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f8c0f4780 0x7f5f8c0f7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:50.751 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.761+0000 7f5f89d9b700 1 -- 172.21.15.23:0/4005100079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f8c0f4780 msgr2=0x7f5f8c0f7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:50.751 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.761+0000 7f5f89d9b700 1 --2- 172.21.15.23:0/4005100079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f8c0f4780 0x7f5f8c0f7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:50.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.761+0000 7f5f89d9b700 1 -- 172.21.15.23:0/4005100079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f74005040 con 0x7f5f8c0f7d90 2021-11-19T08:42:50.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.761+0000 7f5f89d9b700 1 --2- 172.21.15.23:0/4005100079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f8c0f7d90 0x7f5f8c0f59b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f5f74007a70 tx=0x7f5f74009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:50.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.761+0000 7f5f82ffd700 1 -- 172.21.15.23:0/4005100079 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f74018070 con 0x7f5f8c0f7d90 2021-11-19T08:42:50.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.761+0000 7f5f82ffd700 1 -- 172.21.15.23:0/4005100079 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f74009700 con 0x7f5f8c0f7d90 2021-11-19T08:42:50.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.761+0000 7f5f82ffd700 1 -- 172.21.15.23:0/4005100079 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f7401c620 con 0x7f5f8c0f7d90 2021-11-19T08:42:50.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.761+0000 7f5f90f4e700 1 -- 172.21.15.23:0/4005100079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8c0f6260 con 0x7f5f8c0f7d90 2021-11-19T08:42:50.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.761+0000 7f5f90f4e700 1 -- 172.21.15.23:0/4005100079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8c0f67c0 con 0x7f5f8c0f7d90 2021-11-19T08:42:50.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.762+0000 7f5f90f4e700 1 -- 172.21.15.23:0/4005100079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8c027a00 con 0x7f5f8c0f7d90 2021-11-19T08:42:50.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.762+0000 7f5f82ffd700 1 -- 172.21.15.23:0/4005100079 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5f7401d420 con 0x7f5f8c0f7d90 2021-11-19T08:42:50.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.762+0000 7f5f82ffd700 1 --2- 172.21.15.23:0/4005100079 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5f7805c2a0 0x7f5f7805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:50.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.763+0000 7f5f82ffd700 1 -- 172.21.15.23:0/4005100079 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f5f74078060 con 0x7f5f8c0f7d90 2021-11-19T08:42:50.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.763+0000 7f5f8a59c700 1 --2- 172.21.15.23:0/4005100079 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5f7805c2a0 0x7f5f7805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:50.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.763+0000 7f5f8a59c700 1 --2- 172.21.15.23:0/4005100079 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5f7805c2a0 0x7f5f7805e760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f5f8c05f7c0 tx=0x7f5f84008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:50.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.766+0000 7f5f82ffd700 1 -- 172.21.15.23:0/4005100079 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f7404aa70 con 0x7f5f8c0f7d90 2021-11-19T08:42:50.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:50 smithi134 conmon[36363]: cluster 2021-11-19T08:42:48.864955+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:42:50.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:50 smithi134 conmon[36363]: .14164) 264 : cluster [DBG] pgmap v233: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-19T08:42:50.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:50 smithi134 conmon[36363]: cluster 2021-11-19T08:42:49.460869+0000 mon.smithi023 (mon.0) 675 : cluster [DBG] 2021-11-19T08:42:50.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:50 smithi134 conmon[36363]: mds.? [v2:172.21.15.23:6826/2135889102,v1:172.21.15.23:6827/2135889102] up:standby 2021-11-19T08:42:50.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:50 smithi134 conmon[36363]: cluster 2021-11-19T08: 2021-11-19T08:42:50.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:50 smithi134 conmon[36363]: 42:49.460914+0000 mon.smithi023 (mon.0) 676 : cluster [DBG] 2021-11-19T08:42:50.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:50 smithi134 conmon[36363]: fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:rejoin,1=cephfs.smithi023.wazanl=up:rejoin} 2 up:standby 2021-11-19T08:42:50.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:50 smithi134 conmon[36363]: audit 2021-11-19T08:42:49 2021-11-19T08:42:50.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:50 smithi134 conmon[36363]: .611818+0000 mon.smithi023 (mon.0) 677 : audit 2021-11-19T08:42:50.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:50 smithi134 conmon[36363]: [DBG] from='client.? 172.21.15.23:0/3529608456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:50.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:50 smithi023 conmon[33490]: cluster 2021-11-19T08:42:48.864955+0000 mgr.smithi023.ehlpcl (mgr.14164) 264 : cluster [DBG] pgmap v233: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 307 B/s rd, 0 op/s 2021-11-19T08:42:50.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:50 smithi023 conmon[33490]: cluster 2021-11-19T08:42:49.460869+0000 mon.smithi023 (mon.0) 675 : cluster [DBG] mds.? [v2:172.21.15.23:6826/2135889102,v1:172.21.15.23:6827/2135889102] up:standby 2021-11-19T08:42:50.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:50 smithi023 conmon[33490]: cluster 2021-11-19T08:42:50.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:50 smithi023 conmon[33490]: 2021-11-19T 2021-11-19T08:42:50.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:50 smithi023 conmon[33490]: 08:42:49.460914+0000 mon.smithi023 (mon. 2021-11-19T08:42:50.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:50 smithi023 conmon[33490]: 0) 676 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:rejoin,1=cephfs.smithi023.wazanl=up:rejoin} 2 up:standby 2021-11-19T08:42:50.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:50 smithi023 conmon[33490]: audit 2021-11-19T08:42:50.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:50 smithi023 conmon[33490]: 2021-11-19T08:42:49 2021-11-19T08:42:50.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:50 smithi023 conmon[33490]: .611818+0000 mon.smithi023 (mon.0) 677 : 2021-11-19T08:42:50.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:50 smithi023 conmon[33490]: audit [DBG] from='client.? 172.21.15.23:0/3529608456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:50.949 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.965+0000 7f5f90f4e700 1 -- 172.21.15.23:0/4005100079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f8c0fefe0 con 0x7f5f8c0f7d90 2021-11-19T08:42:50.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.965+0000 7f5f82ffd700 1 -- 172.21.15.23:0/4005100079 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f5f7404e090 con 0x7f5f8c0f7d90 2021-11-19T08:42:50.951 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:42:50.951 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":53,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:42:48.445915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24373,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":46,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.134:6825/4111632986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":4111632986},{"type":"v1","addr":"172.21.15.134:6825","nonce":4111632986}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:42:50.952 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.967+0000 7f5f90f4e700 1 -- 172.21.15.23:0/4005100079 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5f7805c2a0 msgr2=0x7f5f7805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:50.953 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.967+0000 7f5f90f4e700 1 --2- 172.21.15.23:0/4005100079 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5f7805c2a0 0x7f5f7805e760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f5f8c05f7c0 tx=0x7f5f84008040).stop 2021-11-19T08:42:50.953 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.967+0000 7f5f90f4e700 1 -- 172.21.15.23:0/4005100079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f8c0f7d90 msgr2=0x7f5f8c0f59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:50.953 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.967+0000 7f5f90f4e700 1 --2- 172.21.15.23:0/4005100079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f8c0f7d90 0x7f5f8c0f59b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f5f74007a70 tx=0x7f5f74009b30).stop 2021-11-19T08:42:50.953 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.967+0000 7f5f90f4e700 1 -- 172.21.15.23:0/4005100079 shutdown_connections 2021-11-19T08:42:50.954 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.967+0000 7f5f90f4e700 1 --2- 172.21.15.23:0/4005100079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f8c0f4780 0x7f5f8c0f7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:50.954 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.967+0000 7f5f90f4e700 1 --2- 172.21.15.23:0/4005100079 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5f7805c2a0 0x7f5f7805e760 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:50.954 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.967+0000 7f5f90f4e700 1 --2- 172.21.15.23:0/4005100079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f8c0f7d90 0x7f5f8c0f59b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:50.954 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.967+0000 7f5f90f4e700 1 -- 172.21.15.23:0/4005100079 >> 172.21.15.23:0/4005100079 conn(0x7f5f8c0f04a0 msgr2=0x7f5f8c0fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:50.955 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.967+0000 7f5f90f4e700 1 -- 172.21.15.23:0/4005100079 shutdown_connections 2021-11-19T08:42:50.955 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:50.967+0000 7f5f90f4e700 1 -- 172.21.15.23:0/4005100079 wait complete. 2021-11-19T08:42:50.955 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 54 2021-11-19T08:42:50.964 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:42:48.445915+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24373, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 46, 'state': 'up:rejoin', 'state_seq': 23, 'addr': '172.21.15.134:6825/4111632986', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 4111632986}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 4111632986}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:42:50.964 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:42:51.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:51 smithi134 conmon[36363]: audit 2021-11-19T08:42:50.966438+0000 mon.smithi023 (mon.0) 678 : audit [DBG] from='client.? 172.21.15.23:0/4005100079' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:51.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:51 smithi023 conmon[33490]: audit 2021-11-19T08:42:50.966438+0000 mon.smithi023 ( 2021-11-19T08:42:51.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:51 smithi023 conmon[33490]: mon.0) 678 : audit [DBG] from='client.? 172.21.15.23:0/4005100079' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:51.965 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:42:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.114+0000 7f3449827700 1 -- 172.21.15.23:0/1408394270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34440f6680 msgr2=0x7f34440f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.114+0000 7f3449827700 1 --2- 172.21.15.23:0/1408394270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34440f6680 0x7f34440f6aa0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f34340099e0 tx=0x7f3434004d10).stop 2021-11-19T08:42:52.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.114+0000 7f3449827700 1 -- 172.21.15.23:0/1408394270 shutdown_connections 2021-11-19T08:42:52.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.114+0000 7f3449827700 1 --2- 172.21.15.23:0/1408394270 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34440f7920 0x7f34440f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:52.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.114+0000 7f3449827700 1 --2- 172.21.15.23:0/1408394270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34440f6680 0x7f34440f6aa0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:52.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.114+0000 7f3449827700 1 -- 172.21.15.23:0/1408394270 >> 172.21.15.23:0/1408394270 conn(0x7f34440f1de0 msgr2=0x7f34440f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:52.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.114+0000 7f3449827700 1 -- 172.21.15.23:0/1408394270 shutdown_connections 2021-11-19T08:42:52.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.114+0000 7f3449827700 1 -- 172.21.15.23:0/1408394270 wait complete. 2021-11-19T08:42:52.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.114+0000 7f3449827700 1 Processor -- start 2021-11-19T08:42:52.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.114+0000 7f3449827700 1 -- start start 2021-11-19T08:42:52.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f3449827700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34440f6680 0x7f344406b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:52.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f3449827700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34440f7920 0x7f3444069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:52.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f3449827700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344406bd70 con 0x7f34440f7920 2021-11-19T08:42:52.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f3449827700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344410ac90 con 0x7f34440f6680 2021-11-19T08:42:52.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f34427fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34440f7920 0x7f3444069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:52.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f34427fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34440f7920 0x7f3444069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47576/0 (socket says 172.21.15.23:47576) 2021-11-19T08:42:52.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f34427fc700 1 -- 172.21.15.23:0/469064847 learned_addr learned my addr 172.21.15.23:0/469064847 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:42:52.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f3442ffd700 1 --2- 172.21.15.23:0/469064847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34440f6680 0x7f344406b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f34427fc700 1 -- 172.21.15.23:0/469064847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34440f6680 msgr2=0x7f344406b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f34427fc700 1 --2- 172.21.15.23:0/469064847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34440f6680 0x7f344406b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f34427fc700 1 -- 172.21.15.23:0/469064847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3434009640 con 0x7f34440f7920 2021-11-19T08:42:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f3442ffd700 1 --2- 172.21.15.23:0/469064847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34440f6680 0x7f344406b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:42:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f34427fc700 1 --2- 172.21.15.23:0/469064847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34440f7920 0x7f3444069e20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f342c002700 tx=0x7f342c009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f343b7fe700 1 -- 172.21.15.23:0/469064847 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f342c009320 con 0x7f34440f7920 2021-11-19T08:42:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f343b7fe700 1 -- 172.21.15.23:0/469064847 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f342c00f040 con 0x7f34440f7920 2021-11-19T08:42:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f3449827700 1 -- 172.21.15.23:0/469064847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344406a310 con 0x7f34440f7920 2021-11-19T08:42:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f343b7fe700 1 -- 172.21.15.23:0/469064847 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f342c014820 con 0x7f34440f7920 2021-11-19T08:42:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.115+0000 7f3449827700 1 -- 172.21.15.23:0/469064847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f344406a850 con 0x7f34440f7920 2021-11-19T08:42:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.116+0000 7f3449827700 1 -- 172.21.15.23:0/469064847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3444046330 con 0x7f34440f7920 2021-11-19T08:42:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.117+0000 7f343b7fe700 1 -- 172.21.15.23:0/469064847 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f342c0077b0 con 0x7f34440f7920 2021-11-19T08:42:52.110 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.117+0000 7f343b7fe700 1 --2- 172.21.15.23:0/469064847 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f343005c2a0 0x7f343005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:52.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.117+0000 7f343b7fe700 1 -- 172.21.15.23:0/469064847 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f342c076410 con 0x7f34440f7920 2021-11-19T08:42:52.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.117+0000 7f3442ffd700 1 --2- 172.21.15.23:0/469064847 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f343005c2a0 0x7f343005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:52.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.118+0000 7f3442ffd700 1 --2- 172.21.15.23:0/469064847 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f343005c2a0 0x7f343005e760 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f343400b3b0 tx=0x7f3434004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:52.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.121+0000 7f343b7fe700 1 -- 172.21.15.23:0/469064847 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f342c048e20 con 0x7f34440f7920 2021-11-19T08:42:52.301 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.317+0000 7f3449827700 1 -- 172.21.15.23:0/469064847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f344406a4a0 con 0x7f34440f7920 2021-11-19T08:42:52.302 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.317+0000 7f343b7fe700 1 -- 172.21.15.23:0/469064847 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f344406a4a0 con 0x7f34440f7920 2021-11-19T08:42:52.303 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:42:52.303 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":53,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:42:48.445915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24373,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":46,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.134:6825/4111632986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":4111632986},{"type":"v1","addr":"172.21.15.134:6825","nonce":4111632986}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:42:52.305 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.319+0000 7f3449827700 1 -- 172.21.15.23:0/469064847 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f343005c2a0 msgr2=0x7f343005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:52.305 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.319+0000 7f3449827700 1 --2- 172.21.15.23:0/469064847 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f343005c2a0 0x7f343005e760 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f343400b3b0 tx=0x7f3434004d10).stop 2021-11-19T08:42:52.307 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.319+0000 7f3449827700 1 -- 172.21.15.23:0/469064847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34440f7920 msgr2=0x7f3444069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:52.308 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.319+0000 7f3449827700 1 --2- 172.21.15.23:0/469064847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34440f7920 0x7f3444069e20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f342c002700 tx=0x7f342c009e00).stop 2021-11-19T08:42:52.308 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.319+0000 7f3449827700 1 -- 172.21.15.23:0/469064847 shutdown_connections 2021-11-19T08:42:52.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.319+0000 7f3449827700 1 --2- 172.21.15.23:0/469064847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f34440f6680 0x7f344406b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.319+0000 7f3449827700 1 --2- 172.21.15.23:0/469064847 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f343005c2a0 0x7f343005e760 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:52.313 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.319+0000 7f3449827700 1 --2- 172.21.15.23:0/469064847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34440f7920 0x7f3444069e20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:52.314 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.319+0000 7f3449827700 1 -- 172.21.15.23:0/469064847 >> 172.21.15.23:0/469064847 conn(0x7f34440f1de0 msgr2=0x7f34440fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:52.314 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.319+0000 7f3449827700 1 -- 172.21.15.23:0/469064847 shutdown_connections 2021-11-19T08:42:52.315 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:52.319+0000 7f3449827700 1 -- 172.21.15.23:0/469064847 wait complete. 2021-11-19T08:42:52.315 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 54 2021-11-19T08:42:52.317 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:42:48.445915+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24373, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 46, 'state': 'up:rejoin', 'state_seq': 23, 'addr': '172.21.15.134:6825/4111632986', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 4111632986}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 4111632986}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:42:52.318 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:42:52.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:52 smithi023 conmon[33490]: cluster 2021-11-19T08:42:50.865487+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:42:52.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:52 smithi023 conmon[33490]: mgr.14164) 265 : cluster [DBG] pgmap v234: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:52.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:52 smithi023 conmon[33490]: audit 2021-11-19T08:42:52.318471+0000 mon.smithi023 (mon.0) 679 : audit [DBG] 2021-11-19T08:42:52.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:52 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/469064847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:52.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:52 smithi134 conmon[36363]: cluster 2021-11-19T08:42:50.865487+0000 mgr.smithi023.ehlpcl (mgr.14164) 265 : cluster [DBG] pgmap v234: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:52.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:52 smithi134 conmon[36363]: audit 2021-11-19T08:42:52.318471+0000 mon.smithi023 (mon.0) 679 : audit [DBG] from='client.? 172.21.15.23:0/469064847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:53.320 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:42:53.454 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.465+0000 7f1715789700 1 -- 172.21.15.23:0/2975988707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17100f5500 msgr2=0x7f17100f5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:53.454 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.465+0000 7f1715789700 1 --2- 172.21.15.23:0/2975988707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17100f5500 0x7f17100f5920 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f1700004660 tx=0x7f1700009b30).stop 2021-11-19T08:42:53.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.465+0000 7f1715789700 1 -- 172.21.15.23:0/2975988707 shutdown_connections 2021-11-19T08:42:53.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.465+0000 7f1715789700 1 --2- 172.21.15.23:0/2975988707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17100f67a0 0x7f17100f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:53.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.465+0000 7f1715789700 1 --2- 172.21.15.23:0/2975988707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17100f5500 0x7f17100f5920 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:53.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.465+0000 7f1715789700 1 -- 172.21.15.23:0/2975988707 >> 172.21.15.23:0/2975988707 conn(0x7f17100f0c60 msgr2=0x7f17100f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:53.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.465+0000 7f1715789700 1 -- 172.21.15.23:0/2975988707 shutdown_connections 2021-11-19T08:42:53.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.465+0000 7f1715789700 1 -- 172.21.15.23:0/2975988707 wait complete. 2021-11-19T08:42:53.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.466+0000 7f1715789700 1 Processor -- start 2021-11-19T08:42:53.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.466+0000 7f1715789700 1 -- start start 2021-11-19T08:42:53.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.466+0000 7f1715789700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17100f5500 0x7f1710109a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:53.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.466+0000 7f1715789700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17100f67a0 0x7f171010bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:53.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.466+0000 7f1715789700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1710060520 con 0x7f17100f5500 2021-11-19T08:42:53.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.466+0000 7f1715789700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1710060660 con 0x7f17100f67a0 2021-11-19T08:42:53.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.466+0000 7f170effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17100f5500 0x7f1710109a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:53.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.466+0000 7f170effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17100f5500 0x7f1710109a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47586/0 (socket says 172.21.15.23:47586) 2021-11-19T08:42:53.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.466+0000 7f170effd700 1 -- 172.21.15.23:0/1051959454 learned_addr learned my addr 172.21.15.23:0/1051959454 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:42:53.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.466+0000 7f170e7fc700 1 --2- 172.21.15.23:0/1051959454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17100f67a0 0x7f171010bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:53.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.467+0000 7f170effd700 1 -- 172.21.15.23:0/1051959454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17100f67a0 msgr2=0x7f171010bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:53.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.467+0000 7f170effd700 1 --2- 172.21.15.23:0/1051959454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17100f67a0 0x7f171010bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:53.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.467+0000 7f170effd700 1 -- 172.21.15.23:0/1051959454 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1700005040 con 0x7f17100f5500 2021-11-19T08:42:53.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.467+0000 7f170e7fc700 1 --2- 172.21.15.23:0/1051959454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17100f67a0 0x7f171010bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:42:53.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.467+0000 7f170effd700 1 --2- 172.21.15.23:0/1051959454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17100f5500 0x7f1710109a60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f1700009f50 tx=0x7f1700009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:53.460 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.467+0000 7f16f77fe700 1 -- 172.21.15.23:0/1051959454 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1700018070 con 0x7f17100f5500 2021-11-19T08:42:53.460 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.467+0000 7f16f77fe700 1 -- 172.21.15.23:0/1051959454 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1700013410 con 0x7f17100f5500 2021-11-19T08:42:53.460 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.467+0000 7f1715789700 1 -- 172.21.15.23:0/1051959454 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171010c4b0 con 0x7f17100f5500 2021-11-19T08:42:53.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.467+0000 7f16f77fe700 1 -- 172.21.15.23:0/1051959454 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f170001c620 con 0x7f17100f5500 2021-11-19T08:42:53.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.467+0000 7f1715789700 1 -- 172.21.15.23:0/1051959454 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171010c9e0 con 0x7f17100f5500 2021-11-19T08:42:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.468+0000 7f1715789700 1 -- 172.21.15.23:0/1051959454 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17100f48f0 con 0x7f17100f5500 2021-11-19T08:42:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.468+0000 7f16f77fe700 1 -- 172.21.15.23:0/1051959454 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f170001d420 con 0x7f17100f5500 2021-11-19T08:42:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.469+0000 7f16f77fe700 1 --2- 172.21.15.23:0/1051959454 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f16f805c300 0x7f16f805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.469+0000 7f16f77fe700 1 -- 172.21.15.23:0/1051959454 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f1700077f00 con 0x7f17100f5500 2021-11-19T08:42:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.469+0000 7f170e7fc700 1 --2- 172.21.15.23:0/1051959454 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f16f805c300 0x7f16f805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.469+0000 7f170e7fc700 1 --2- 172.21.15.23:0/1051959454 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f16f805c300 0x7f16f805e7c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f17040041d0 tx=0x7f1704009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.473+0000 7f16f77fe700 1 -- 172.21.15.23:0/1051959454 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f170004a990 con 0x7f17100f5500 2021-11-19T08:42:53.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.667+0000 7f1715789700 1 -- 172.21.15.23:0/1051959454 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1710045ad0 con 0x7f17100f5500 2021-11-19T08:42:53.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.667+0000 7f16f77fe700 1 -- 172.21.15.23:0/1051959454 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f170004dfb0 con 0x7f17100f5500 2021-11-19T08:42:53.652 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:42:53.653 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":53,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:42:48.445915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24373,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":46,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.134:6825/4111632986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":4111632986},{"type":"v1","addr":"172.21.15.134:6825","nonce":4111632986}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:42:53.654 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.669+0000 7f1715789700 1 -- 172.21.15.23:0/1051959454 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f16f805c300 msgr2=0x7f16f805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:53.654 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.669+0000 7f1715789700 1 --2- 172.21.15.23:0/1051959454 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f16f805c300 0x7f16f805e7c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f17040041d0 tx=0x7f1704009040).stop 2021-11-19T08:42:53.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.669+0000 7f1715789700 1 -- 172.21.15.23:0/1051959454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17100f5500 msgr2=0x7f1710109a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:53.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.669+0000 7f1715789700 1 --2- 172.21.15.23:0/1051959454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17100f5500 0x7f1710109a60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f1700009f50 tx=0x7f1700009b30).stop 2021-11-19T08:42:53.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.669+0000 7f1715789700 1 -- 172.21.15.23:0/1051959454 shutdown_connections 2021-11-19T08:42:53.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.669+0000 7f1715789700 1 --2- 172.21.15.23:0/1051959454 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f16f805c300 0x7f16f805e7c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:53.656 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.669+0000 7f1715789700 1 --2- 172.21.15.23:0/1051959454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17100f5500 0x7f1710109a60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:53.656 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.669+0000 7f1715789700 1 --2- 172.21.15.23:0/1051959454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17100f67a0 0x7f171010bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:53.656 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.669+0000 7f1715789700 1 -- 172.21.15.23:0/1051959454 >> 172.21.15.23:0/1051959454 conn(0x7f17100f0c60 msgr2=0x7f17100fa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:53.656 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.669+0000 7f1715789700 1 -- 172.21.15.23:0/1051959454 shutdown_connections 2021-11-19T08:42:53.657 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:53.669+0000 7f1715789700 1 -- 172.21.15.23:0/1051959454 wait complete. 2021-11-19T08:42:53.657 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 54 2021-11-19T08:42:53.665 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:42:48.445915+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24373, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 46, 'state': 'up:rejoin', 'state_seq': 23, 'addr': '172.21.15.134:6825/4111632986', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 4111632986}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 4111632986}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:42:53.666 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:42:54.667 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:42:54.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:54 smithi134 conmon[36363]: cluster 2021-11-19T08:42:52.866015+0000 mgr.smithi023.ehlpcl (mgr.14164) 266 : cluster [DBG] pgmap v235: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:54.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:54 smithi134 conmon[36363]: 2021-11-19T08:42:54.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:54 smithi134 conmon[36363]: audit 2021-11-19T08:42:53.668150+0000 mon.smithi023 (mon.0) 680 : audit [DBG] from='client.? 172.21.15.23:0/1051959454' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:54.801 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.814+0000 7fb9c094c700 1 -- 172.21.15.23:0/2931338565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc0f54d0 msgr2=0x7fb9bc0f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:54.801 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.814+0000 7fb9c094c700 1 --2- 172.21.15.23:0/2931338565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc0f54d0 0x7fb9bc0f58f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fb9ac004660 tx=0x7fb9ac009b30).stop 2021-11-19T08:42:54.801 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.814+0000 7fb9c094c700 1 -- 172.21.15.23:0/2931338565 shutdown_connections 2021-11-19T08:42:54.801 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.814+0000 7fb9c094c700 1 --2- 172.21.15.23:0/2931338565 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc0f6770 0x7fb9bc0f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-19T08:42:54.802 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.814+0000 7fb9c094c700 1 --2- 172.21.15.23:0/2931338565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc0f54d0 0x7fb9bc0f58f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:54.802 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.814+0000 7fb9c094c700 1 -- 172.21.15.23:0/2931338565 >> 172.21.15.23:0/2931338565 conn(0x7fb9bc0f0c30 msgr2=0x7fb9bc0f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:54.802 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.814+0000 7fb9c094c700 1 -- 172.21.15.23:0/2931338565 shutdown_connections 2021-11-19T08:42:54.802 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.814+0000 7fb9c094c700 1 -- 172.21.15.23:0/2931338565 wait complete. 2021-11-19T08:42:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.815+0000 7fb9c094c700 1 Processor -- start 2021-11-19T08:42:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.815+0000 7fb9c094c700 1 -- start start 2021-11-19T08:42:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.815+0000 7fb9c094c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc0f54d0 0x7fb9bc106db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.815+0000 7fb9c094c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc0f6770 0x7fb9bc1072a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:54.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.815+0000 7fb9c094c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc107930 con 0x7fb9bc0f54d0 2021-11-19T08:42:54.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.815+0000 7fb9c094c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc1080e0 con 0x7fb9bc0f6770 2021-11-19T08:42:54.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.815+0000 7fb9bad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc0f54d0 0x7fb9bc106db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:54.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.815+0000 7fb9bad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc0f54d0 0x7fb9bc106db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47598/0 (socket says 172.21.15.23:47598) 2021-11-19T08:42:54.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.815+0000 7fb9bad9d700 1 -- 172.21.15.23:0/2917014455 learned_addr learned my addr 172.21.15.23:0/2917014455 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:42:54.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.816+0000 7fb9ba59c700 1 --2- 172.21.15.23:0/2917014455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc0f6770 0x7fb9bc1072a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:54.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.816+0000 7fb9bad9d700 1 -- 172.21.15.23:0/2917014455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc0f6770 msgr2=0x7fb9bc1072a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:54.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.816+0000 7fb9bad9d700 1 --2- 172.21.15.23:0/2917014455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc0f6770 0x7fb9bc1072a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:54.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.816+0000 7fb9bad9d700 1 -- 172.21.15.23:0/2917014455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ac005040 con 0x7fb9bc0f54d0 2021-11-19T08:42:54.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.816+0000 7fb9ba59c700 1 --2- 172.21.15.23:0/2917014455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc0f6770 0x7fb9bc1072a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:42:54.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.816+0000 7fb9bad9d700 1 --2- 172.21.15.23:0/2917014455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc0f54d0 0x7fb9bc106db0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fb9ac009f50 tx=0x7fb9ac009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:54.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.816+0000 7fb9a37fe700 1 -- 172.21.15.23:0/2917014455 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9ac018070 con 0x7fb9bc0f54d0 2021-11-19T08:42:54.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.816+0000 7fb9c094c700 1 -- 172.21.15.23:0/2917014455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9bc108360 con 0x7fb9bc0f54d0 2021-11-19T08:42:54.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.816+0000 7fb9a37fe700 1 -- 172.21.15.23:0/2917014455 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9ac013410 con 0x7fb9bc0f54d0 2021-11-19T08:42:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.816+0000 7fb9a37fe700 1 -- 172.21.15.23:0/2917014455 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9ac01c620 con 0x7fb9bc0f54d0 2021-11-19T08:42:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.816+0000 7fb9c094c700 1 -- 172.21.15.23:0/2917014455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9bc1088c0 con 0x7fb9bc0f54d0 2021-11-19T08:42:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.817+0000 7fb9c094c700 1 -- 172.21.15.23:0/2917014455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9bc0f48c0 con 0x7fb9bc0f54d0 2021-11-19T08:42:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.817+0000 7fb9a37fe700 1 -- 172.21.15.23:0/2917014455 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb9ac01d420 con 0x7fb9bc0f54d0 2021-11-19T08:42:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.818+0000 7fb9a37fe700 1 --2- 172.21.15.23:0/2917014455 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb9a405c300 0x7fb9a405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.818+0000 7fb9a37fe700 1 -- 172.21.15.23:0/2917014455 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fb9ac077fe0 con 0x7fb9bc0f54d0 2021-11-19T08:42:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.818+0000 7fb9ba59c700 1 --2- 172.21.15.23:0/2917014455 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb9a405c300 0x7fb9a405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.818+0000 7fb9ba59c700 1 --2- 172.21.15.23:0/2917014455 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb9a405c300 0x7fb9a405e7c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fb9bc106b40 tx=0x7fb9b0008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:54.822+0000 7fb9a37fe700 1 -- 172.21.15.23:0/2917014455 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9ac04a9f0 con 0x7fb9bc0f54d0 2021-11-19T08:42:54.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:54 smithi023 conmon[33490]: cluster 2021-11-19T08:42:52.866015+0000 mgr.smithi023.ehlpcl 2021-11-19T08:42:54.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:54 smithi023 conmon[33490]: (mgr.14164) 266 : cluster [DBG] pgmap v235: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:54.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:54 smithi023 conmon[33490]: audit 2021-11-19T08:42:53.668150+0000 mon.smithi023 (mon.0) 680 : audit [DBG] from='client.? 172.21.15.23:0/1051959454' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:55.003 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:55.018+0000 7fb9c094c700 1 -- 172.21.15.23:0/2917014455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9bc108f00 con 0x7fb9bc0f54d0 2021-11-19T08:42:55.003 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:55.019+0000 7fb9a37fe700 1 -- 172.21.15.23:0/2917014455 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2951 (secure 0 0 0) 0x7fb9ac04e010 con 0x7fb9bc0f54d0 2021-11-19T08:42:55.004 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:42:55.004 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":53,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:42:48.445915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24373,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":46,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.134:6825/4111632986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":4111632986},{"type":"v1","addr":"172.21.15.134:6825","nonce":4111632986}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:42:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:55.020+0000 7fb9c094c700 1 -- 172.21.15.23:0/2917014455 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb9a405c300 msgr2=0x7fb9a405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:55.020+0000 7fb9c094c700 1 --2- 172.21.15.23:0/2917014455 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb9a405c300 0x7fb9a405e7c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fb9bc106b40 tx=0x7fb9b0008040).stop 2021-11-19T08:42:55.006 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:55.020+0000 7fb9c094c700 1 -- 172.21.15.23:0/2917014455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc0f54d0 msgr2=0x7fb9bc106db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:55.006 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:55.020+0000 7fb9c094c700 1 --2- 172.21.15.23:0/2917014455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc0f54d0 0x7fb9bc106db0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fb9ac009f50 tx=0x7fb9ac009b30).stop 2021-11-19T08:42:55.006 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:55.020+0000 7fb9c094c700 1 -- 172.21.15.23:0/2917014455 shutdown_connections 2021-11-19T08:42:55.006 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:55.020+0000 7fb9c094c700 1 --2- 172.21.15.23:0/2917014455 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb9a405c300 0x7fb9a405e7c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:55.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:55.020+0000 7fb9c094c700 1 --2- 172.21.15.23:0/2917014455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc0f54d0 0x7fb9bc106db0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:55.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:55.020+0000 7fb9c094c700 1 --2- 172.21.15.23:0/2917014455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9bc0f6770 0x7fb9bc1072a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:55.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:55.020+0000 7fb9c094c700 1 -- 172.21.15.23:0/2917014455 >> 172.21.15.23:0/2917014455 conn(0x7fb9bc0f0c30 msgr2=0x7fb9bc05f780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:55.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:55.021+0000 7fb9c094c700 1 -- 172.21.15.23:0/2917014455 shutdown_connections 2021-11-19T08:42:55.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:55.021+0000 7fb9c094c700 1 -- 172.21.15.23:0/2917014455 wait complete. 2021-11-19T08:42:55.008 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 54 2021-11-19T08:42:55.018 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:42:48.445915+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24373, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 46, 'state': 'up:rejoin', 'state_seq': 23, 'addr': '172.21.15.134:6825/4111632986', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 4111632986}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 4111632986}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:42:55.018 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:42:55.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:55 smithi134 conmon[36363]: audit 2021-11-19T08:42:55.019739+0000 mon.smithi023 (mon.0) 681 : audit [DBG] from='client.? 172.21.15.23:0/2917014455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:55.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:55 smithi134 conmon[36363]: 2021-11-19T08:42:55.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:55 smithi023 conmon[33490]: audit 2021-11-19T08:42:55.019739+0000 mon.smithi023 (mon 2021-11-19T08:42:55.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:55 smithi023 conmon[33490]: .0) 681 : audit [DBG] from='client.? 172.21.15.23:0/2917014455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:56.020 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:42:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.167+0000 7fb45a6a0700 1 -- 172.21.15.23:0/1544582332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4540f7920 msgr2=0x7fb4540f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.167+0000 7fb45a6a0700 1 --2- 172.21.15.23:0/1544582332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4540f7920 0x7fb4540f7d80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fb448004660 tx=0x7fb448009b30).stop 2021-11-19T08:42:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.167+0000 7fb45a6a0700 1 -- 172.21.15.23:0/1544582332 shutdown_connections 2021-11-19T08:42:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.167+0000 7fb45a6a0700 1 --2- 172.21.15.23:0/1544582332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4540f7920 0x7fb4540f7d80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.167+0000 7fb45a6a0700 1 --2- 172.21.15.23:0/1544582332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb4540f6680 0x7fb4540f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.167+0000 7fb45a6a0700 1 -- 172.21.15.23:0/1544582332 >> 172.21.15.23:0/1544582332 conn(0x7fb4540f1de0 msgr2=0x7fb4540f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.168+0000 7fb45a6a0700 1 -- 172.21.15.23:0/1544582332 shutdown_connections 2021-11-19T08:42:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.168+0000 7fb45a6a0700 1 -- 172.21.15.23:0/1544582332 wait complete. 2021-11-19T08:42:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.168+0000 7fb45a6a0700 1 Processor -- start 2021-11-19T08:42:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.168+0000 7fb45a6a0700 1 -- start start 2021-11-19T08:42:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.168+0000 7fb45a6a0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4540f6680 0x7fb45406b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.168+0000 7fb45a6a0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb4540f7920 0x7fb454069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.168+0000 7fb45a6a0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45406bd70 con 0x7fb4540f6680 2021-11-19T08:42:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.168+0000 7fb45a6a0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45406a310 con 0x7fb4540f7920 2021-11-19T08:42:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.169+0000 7fb453fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4540f6680 0x7fb45406b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.169+0000 7fb453fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4540f6680 0x7fb45406b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47606/0 (socket says 172.21.15.23:47606) 2021-11-19T08:42:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.169+0000 7fb453fff700 1 -- 172.21.15.23:0/2397461781 learned_addr learned my addr 172.21.15.23:0/2397461781 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:42:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.169+0000 7fb4537fe700 1 --2- 172.21.15.23:0/2397461781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb4540f7920 0x7fb454069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.169+0000 7fb453fff700 1 -- 172.21.15.23:0/2397461781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb4540f7920 msgr2=0x7fb454069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:56.159 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.169+0000 7fb453fff700 1 --2- 172.21.15.23:0/2397461781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb4540f7920 0x7fb454069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:56.159 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.169+0000 7fb453fff700 1 -- 172.21.15.23:0/2397461781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb448005040 con 0x7fb4540f6680 2021-11-19T08:42:56.159 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.169+0000 7fb4537fe700 1 --2- 172.21.15.23:0/2397461781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb4540f7920 0x7fb454069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:42:56.159 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.169+0000 7fb453fff700 1 --2- 172.21.15.23:0/2397461781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4540f6680 0x7fb45406b7c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fb444002700 tx=0x7fb444009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.169+0000 7fb450ff9700 1 -- 172.21.15.23:0/2397461781 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb444009320 con 0x7fb4540f6680 2021-11-19T08:42:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.169+0000 7fb450ff9700 1 -- 172.21.15.23:0/2397461781 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb44400f040 con 0x7fb4540f6680 2021-11-19T08:42:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.169+0000 7fb45a6a0700 1 -- 172.21.15.23:0/2397461781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45406a590 con 0x7fb4540f6680 2021-11-19T08:42:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.169+0000 7fb450ff9700 1 -- 172.21.15.23:0/2397461781 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb444014900 con 0x7fb4540f6680 2021-11-19T08:42:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.169+0000 7fb45a6a0700 1 -- 172.21.15.23:0/2397461781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45406ab50 con 0x7fb4540f6680 2021-11-19T08:42:56.162 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.170+0000 7fb45a6a0700 1 -- 172.21.15.23:0/2397461781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb454027a00 con 0x7fb4540f6680 2021-11-19T08:42:56.162 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.171+0000 7fb450ff9700 1 -- 172.21.15.23:0/2397461781 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb444009480 con 0x7fb4540f6680 2021-11-19T08:42:56.162 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.171+0000 7fb450ff9700 1 --2- 172.21.15.23:0/2397461781 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb43c05c2a0 0x7fb43c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.171+0000 7fb450ff9700 1 -- 172.21.15.23:0/2397461781 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fb4440764c0 con 0x7fb4540f6680 2021-11-19T08:42:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.171+0000 7fb4537fe700 1 --2- 172.21.15.23:0/2397461781 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb43c05c2a0 0x7fb43c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.171+0000 7fb4537fe700 1 --2- 172.21.15.23:0/2397461781 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb43c05c2a0 0x7fb43c05e760 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fb448005010 tx=0x7fb448009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.175+0000 7fb450ff9700 1 -- 172.21.15.23:0/2397461781 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb444048ed0 con 0x7fb4540f6680 2021-11-19T08:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.369+0000 7fb45a6a0700 1 -- 172.21.15.23:0/2397461781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4540fef00 con 0x7fb4540f6680 2021-11-19T08:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.369+0000 7fb450ff9700 1 -- 172.21.15.23:0/2397461781 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2951 (secure 0 0 0) 0x7fb44404c4f0 con 0x7fb4540f6680 2021-11-19T08:42:56.354 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:42:56.355 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":53,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:42:48.445915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24373,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":46,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.134:6825/4111632986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":4111632986},{"type":"v1","addr":"172.21.15.134:6825","nonce":4111632986}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.371+0000 7fb45a6a0700 1 -- 172.21.15.23:0/2397461781 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb43c05c2a0 msgr2=0x7fb43c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.371+0000 7fb45a6a0700 1 --2- 172.21.15.23:0/2397461781 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb43c05c2a0 0x7fb43c05e760 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fb448005010 tx=0x7fb448009b30).stop 2021-11-19T08:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.371+0000 7fb45a6a0700 1 -- 172.21.15.23:0/2397461781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4540f6680 msgr2=0x7fb45406b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.371+0000 7fb45a6a0700 1 --2- 172.21.15.23:0/2397461781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4540f6680 0x7fb45406b7c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fb444002700 tx=0x7fb444009d80).stop 2021-11-19T08:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.371+0000 7fb45a6a0700 1 -- 172.21.15.23:0/2397461781 shutdown_connections 2021-11-19T08:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.371+0000 7fb45a6a0700 1 --2- 172.21.15.23:0/2397461781 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb43c05c2a0 0x7fb43c05e760 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.371+0000 7fb45a6a0700 1 --2- 172.21.15.23:0/2397461781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4540f6680 0x7fb45406b7c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.371+0000 7fb45a6a0700 1 --2- 172.21.15.23:0/2397461781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb4540f7920 0x7fb454069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:56.358 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.371+0000 7fb45a6a0700 1 -- 172.21.15.23:0/2397461781 >> 172.21.15.23:0/2397461781 conn(0x7fb4540f1de0 msgr2=0x7fb4540fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:56.358 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.371+0000 7fb45a6a0700 1 -- 172.21.15.23:0/2397461781 shutdown_connections 2021-11-19T08:42:56.358 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:56.371+0000 7fb45a6a0700 1 -- 172.21.15.23:0/2397461781 wait complete. 2021-11-19T08:42:56.358 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 54 2021-11-19T08:42:56.368 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:42:48.445915+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24373, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 46, 'state': 'up:rejoin', 'state_seq': 23, 'addr': '172.21.15.134:6825/4111632986', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 4111632986}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 4111632986}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:42:56.368 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:42:56.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:56 smithi134 conmon[36363]: cluster 2021-11-19T08:42:54.866537+0000 mgr.smithi023.ehlpcl (mgr.14164) 267 : cluster [DBG] pgmap v236: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:56.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:56 smithi134 conmon[36363]: audit 2021-11- 2021-11-19T08:42:56.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:56 smithi134 conmon[36363]: 19T08:42:56.370361+0000 mon.smithi023 (mon.0) 682 : audit [DBG] from='client.? 172.21.15.23:0/2397461781' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:56.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:56 smithi023 conmon[33490]: cluster 2021-11-19T08:42:54.866537+0000 mgr.smithi023.ehlpcl (mgr.14164) 267 : cluster [DBG] pgmap v236: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:42:56.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:56 smithi023 conmon[33490]: audit 2021-11-19T08:42:56.370361+0000 mon.smithi023 (mon.0) 682 : audit [DBG] from='client.? 172.21.15.23:0/2397461781' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:57.369 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:42:57.504 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.517+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/2491827983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c880f5570 msgr2=0x7f3c880f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.517+0000 7f3c8d6c2700 1 --2- 172.21.15.23:0/2491827983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c880f5570 0x7f3c880f5990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f3c78004660 tx=0x7f3c78009b30).stop 2021-11-19T08:42:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.517+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/2491827983 shutdown_connections 2021-11-19T08:42:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.517+0000 7f3c8d6c2700 1 --2- 172.21.15.23:0/2491827983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c880f6810 0x7f3c880f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-19T08:42:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.517+0000 7f3c8d6c2700 1 --2- 172.21.15.23:0/2491827983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c880f5570 0x7f3c880f5990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.517+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/2491827983 >> 172.21.15.23:0/2491827983 conn(0x7f3c880f0cb0 msgr2=0x7f3c880f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.517+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/2491827983 shutdown_connections 2021-11-19T08:42:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.517+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/2491827983 wait complete. 2021-11-19T08:42:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.518+0000 7f3c8d6c2700 1 Processor -- start 2021-11-19T08:42:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.518+0000 7f3c8d6c2700 1 -- start start 2021-11-19T08:42:57.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.518+0000 7f3c8d6c2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c880f5570 0x7f3c88060570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:57.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.518+0000 7f3c8d6c2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c880f6810 0x7f3c8810bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:57.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.518+0000 7f3c8d6c2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8810c1a0 con 0x7f3c880f5570 2021-11-19T08:42:57.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.518+0000 7f3c8d6c2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8810c2e0 con 0x7f3c880f6810 2021-11-19T08:42:57.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.518+0000 7f3c86ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c880f5570 0x7f3c88060570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:57.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.518+0000 7f3c86ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c880f5570 0x7f3c88060570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47618/0 (socket says 172.21.15.23:47618) 2021-11-19T08:42:57.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.518+0000 7f3c86ffd700 1 -- 172.21.15.23:0/1417505693 learned_addr learned my addr 172.21.15.23:0/1417505693 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:42:57.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.519+0000 7f3c867fc700 1 --2- 172.21.15.23:0/1417505693 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c880f6810 0x7f3c8810bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:57.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.519+0000 7f3c86ffd700 1 -- 172.21.15.23:0/1417505693 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c880f6810 msgr2=0x7f3c8810bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:57.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.519+0000 7f3c86ffd700 1 --2- 172.21.15.23:0/1417505693 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c880f6810 0x7f3c8810bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.519+0000 7f3c86ffd700 1 -- 172.21.15.23:0/1417505693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c78005040 con 0x7f3c880f5570 2021-11-19T08:42:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.519+0000 7f3c867fc700 1 --2- 172.21.15.23:0/1417505693 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c880f6810 0x7f3c8810bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:42:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.519+0000 7f3c86ffd700 1 --2- 172.21.15.23:0/1417505693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c880f5570 0x7f3c88060570 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f3c78009f50 tx=0x7f3c78009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.519+0000 7f3c6f7fe700 1 -- 172.21.15.23:0/1417505693 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c78018070 con 0x7f3c880f5570 2021-11-19T08:42:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.519+0000 7f3c6f7fe700 1 -- 172.21.15.23:0/1417505693 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c78013410 con 0x7f3c880f5570 2021-11-19T08:42:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.519+0000 7f3c6f7fe700 1 -- 172.21.15.23:0/1417505693 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c7801c620 con 0x7f3c880f5570 2021-11-19T08:42:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.519+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/1417505693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8810c560 con 0x7f3c880f5570 2021-11-19T08:42:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.519+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/1417505693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8810cac0 con 0x7f3c880f5570 2021-11-19T08:42:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.520+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/1417505693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c88027a00 con 0x7f3c880f5570 2021-11-19T08:42:57.513 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.520+0000 7f3c6f7fe700 1 -- 172.21.15.23:0/1417505693 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3c7801d420 con 0x7f3c880f5570 2021-11-19T08:42:57.513 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.521+0000 7f3c6f7fe700 1 --2- 172.21.15.23:0/1417505693 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3c7005c2a0 0x7f3c7005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:57.517 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.521+0000 7f3c6f7fe700 1 -- 172.21.15.23:0/1417505693 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f3c78077fc0 con 0x7f3c880f5570 2021-11-19T08:42:57.517 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.521+0000 7f3c867fc700 1 --2- 172.21.15.23:0/1417505693 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3c7005c2a0 0x7f3c7005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:57.518 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.521+0000 7f3c867fc700 1 --2- 172.21.15.23:0/1417505693 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3c7005c2a0 0x7f3c7005e760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f3c7c00b830 tx=0x7f3c7c008040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:57.518 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.525+0000 7f3c6f7fe700 1 -- 172.21.15.23:0/1417505693 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c780a42b0 con 0x7f3c880f5570 2021-11-19T08:42:57.703 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.718+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/1417505693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c880fdd80 con 0x7f3c880f5570 2021-11-19T08:42:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.719+0000 7f3c6f7fe700 1 -- 172.21.15.23:0/1417505693 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f3c7804a9d0 con 0x7f3c880f5570 2021-11-19T08:42:57.704 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:42:57.704 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":53,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:42:48.445915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24373,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":46,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.134:6825/4111632986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":4111632986},{"type":"v1","addr":"172.21.15.134:6825","nonce":4111632986}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:42:57.705 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.721+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/1417505693 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3c7005c2a0 msgr2=0x7f3c7005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:57.705 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.721+0000 7f3c8d6c2700 1 --2- 172.21.15.23:0/1417505693 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3c7005c2a0 0x7f3c7005e760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f3c7c00b830 tx=0x7f3c7c008040).stop 2021-11-19T08:42:57.706 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.721+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/1417505693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c880f5570 msgr2=0x7f3c88060570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:57.706 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.721+0000 7f3c8d6c2700 1 --2- 172.21.15.23:0/1417505693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c880f5570 0x7f3c88060570 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f3c78009f50 tx=0x7f3c78009b30).stop 2021-11-19T08:42:57.706 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.721+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/1417505693 shutdown_connections 2021-11-19T08:42:57.706 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.721+0000 7f3c8d6c2700 1 --2- 172.21.15.23:0/1417505693 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3c7005c2a0 0x7f3c7005e760 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:57.707 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.721+0000 7f3c8d6c2700 1 --2- 172.21.15.23:0/1417505693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c880f5570 0x7f3c88060570 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:57.707 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.721+0000 7f3c8d6c2700 1 --2- 172.21.15.23:0/1417505693 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c880f6810 0x7f3c8810bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:57.707 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.721+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/1417505693 >> 172.21.15.23:0/1417505693 conn(0x7f3c880f0cb0 msgr2=0x7f3c880fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:57.707 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.721+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/1417505693 shutdown_connections 2021-11-19T08:42:57.707 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:57.721+0000 7f3c8d6c2700 1 -- 172.21.15.23:0/1417505693 wait complete. 2021-11-19T08:42:57.708 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 54 2021-11-19T08:42:57.718 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:42:48.445915+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24373, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 46, 'state': 'up:rejoin', 'state_seq': 23, 'addr': '172.21.15.134:6825/4111632986', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 4111632986}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 4111632986}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:42:57.718 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:42:58.719 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:42:58.748 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:58 smithi023 conmon[33490]: cluster 2021-11-19T08:42:56.866994+0000 mgr.smithi023.ehlpcl (mgr.14164) 268 : cluster [DBG] pgmap v237: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 0 op/s 2021-11-19T08:42:58.748 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:58 smithi023 conmon[33490]: audit 2021-11-19T08:42:57.720093+0000 mon.smithi023 (mon.0) 683 : audit [DBG] from='client.? 172.21.15.23:0/1417505693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:58.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:58 smithi134 conmon[36363]: cluster 2021-11-19T08:42:56.866994+0000 mgr.smithi023.ehlpcl (mgr.14164) 268 : 2021-11-19T08:42:58.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:58 smithi134 conmon[36363]: cluster [DBG] pgmap v237: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 0 op/s 2021-11-19T08:42:58.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:58 smithi134 conmon[36363]: audit 2021-11-19T08:42:57.720093+0000 mon.smithi023 (mon.0) 683 : audit [DBG] from='client.? 172.21.15.23:0/1417505693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:58.855 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.867+0000 7fbf3668f700 1 -- 172.21.15.23:0/1313279086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf300f4a00 msgr2=0x7fbf300f4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:58.855 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.867+0000 7fbf3668f700 1 --2- 172.21.15.23:0/1313279086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf300f4a00 0x7fbf300f4e60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fbf18004660 tx=0x7fbf18009b30).stop 2021-11-19T08:42:58.855 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.867+0000 7fbf3668f700 1 -- 172.21.15.23:0/1313279086 shutdown_connections 2021-11-19T08:42:58.856 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.867+0000 7fbf3668f700 1 --2- 172.21.15.23:0/1313279086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf300f4a00 0x7fbf300f4e60 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:58.856 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.867+0000 7fbf3668f700 1 --2- 172.21.15.23:0/1313279086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf300f63f0 0x7fbf300f6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:58.856 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.867+0000 7fbf3668f700 1 -- 172.21.15.23:0/1313279086 >> 172.21.15.23:0/1313279086 conn(0x7fbf300f04a0 msgr2=0x7fbf300f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:58.856 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.867+0000 7fbf3668f700 1 -- 172.21.15.23:0/1313279086 shutdown_connections 2021-11-19T08:42:58.857 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.867+0000 7fbf3668f700 1 -- 172.21.15.23:0/1313279086 wait complete. 2021-11-19T08:42:58.858 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.868+0000 7fbf3668f700 1 Processor -- start 2021-11-19T08:42:58.858 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.868+0000 7fbf3668f700 1 -- start start 2021-11-19T08:42:58.858 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.868+0000 7fbf3668f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf300f4a00 0x7fbf30105e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:58.858 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.868+0000 7fbf3668f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf300f63f0 0x7fbf30106340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:58.859 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.868+0000 7fbf3668f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf301068f0 con 0x7fbf300f63f0 2021-11-19T08:42:58.859 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.868+0000 7fbf3668f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf30106a30 con 0x7fbf300f4a00 2021-11-19T08:42:58.859 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.868+0000 7fbf2f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf300f63f0 0x7fbf30106340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:58.860 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.869+0000 7fbf2f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf300f63f0 0x7fbf30106340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47630/0 (socket says 172.21.15.23:47630) 2021-11-19T08:42:58.860 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.869+0000 7fbf2f7fe700 1 -- 172.21.15.23:0/3001791536 learned_addr learned my addr 172.21.15.23:0/3001791536 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:42:58.860 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.869+0000 7fbf2ffff700 1 --2- 172.21.15.23:0/3001791536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf300f4a00 0x7fbf30105e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:58.861 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.869+0000 7fbf2f7fe700 1 -- 172.21.15.23:0/3001791536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf300f4a00 msgr2=0x7fbf30105e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:58.861 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.869+0000 7fbf2f7fe700 1 --2- 172.21.15.23:0/3001791536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf300f4a00 0x7fbf30105e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:58.861 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.869+0000 7fbf2f7fe700 1 -- 172.21.15.23:0/3001791536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf18005040 con 0x7fbf300f63f0 2021-11-19T08:42:58.862 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.869+0000 7fbf2f7fe700 1 --2- 172.21.15.23:0/3001791536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf300f63f0 0x7fbf30106340 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fbf18009830 tx=0x7fbf18009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:58.862 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.869+0000 7fbf2cff9700 1 -- 172.21.15.23:0/3001791536 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf1801a070 con 0x7fbf300f63f0 2021-11-19T08:42:58.862 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.869+0000 7fbf2cff9700 1 -- 172.21.15.23:0/3001791536 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf18013410 con 0x7fbf300f63f0 2021-11-19T08:42:58.862 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.869+0000 7fbf3668f700 1 -- 172.21.15.23:0/3001791536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf300f7200 con 0x7fbf300f63f0 2021-11-19T08:42:58.863 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.869+0000 7fbf2cff9700 1 -- 172.21.15.23:0/3001791536 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf1801f880 con 0x7fbf300f63f0 2021-11-19T08:42:58.863 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.869+0000 7fbf3668f700 1 -- 172.21.15.23:0/3001791536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf300f77c0 con 0x7fbf300f63f0 2021-11-19T08:42:58.863 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.870+0000 7fbf3668f700 1 -- 172.21.15.23:0/3001791536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf300fd5b0 con 0x7fbf300f63f0 2021-11-19T08:42:58.864 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.870+0000 7fbf2cff9700 1 -- 172.21.15.23:0/3001791536 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbf18012c10 con 0x7fbf300f63f0 2021-11-19T08:42:58.865 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.871+0000 7fbf2cff9700 1 --2- 172.21.15.23:0/3001791536 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbf1c05c2a0 0x7fbf1c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:42:58.866 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.871+0000 7fbf2cff9700 1 -- 172.21.15.23:0/3001791536 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7fbf1807a2d0 con 0x7fbf300f63f0 2021-11-19T08:42:58.867 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.871+0000 7fbf2ffff700 1 --2- 172.21.15.23:0/3001791536 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbf1c05c2a0 0x7fbf1c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:42:58.868 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.871+0000 7fbf2ffff700 1 --2- 172.21.15.23:0/3001791536 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbf1c05c2a0 0x7fbf1c05e760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fbf300f5e40 tx=0x7fbf2000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:42:58.868 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:58.875+0000 7fbf2cff9700 1 -- 172.21.15.23:0/3001791536 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf1804cce0 con 0x7fbf300f63f0 2021-11-19T08:42:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:59.074+0000 7fbf3668f700 1 -- 172.21.15.23:0/3001791536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf300f80a0 con 0x7fbf300f63f0 2021-11-19T08:42:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:59.074+0000 7fbf2cff9700 1 -- 172.21.15.23:0/3001791536 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2951 (secure 0 0 0) 0x7fbf18050300 con 0x7fbf300f63f0 2021-11-19T08:42:59.059 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:42:59.060 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":53,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:42:48.445915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24373,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":46,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.134:6825/4111632986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":4111632986},{"type":"v1","addr":"172.21.15.134:6825","nonce":4111632986}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:42:59.061 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:59.076+0000 7fbf3668f700 1 -- 172.21.15.23:0/3001791536 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbf1c05c2a0 msgr2=0x7fbf1c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:59.061 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:59.076+0000 7fbf3668f700 1 --2- 172.21.15.23:0/3001791536 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbf1c05c2a0 0x7fbf1c05e760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fbf300f5e40 tx=0x7fbf2000b040).stop 2021-11-19T08:42:59.061 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:59.076+0000 7fbf3668f700 1 -- 172.21.15.23:0/3001791536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf300f63f0 msgr2=0x7fbf30106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:42:59.061 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:59.076+0000 7fbf3668f700 1 --2- 172.21.15.23:0/3001791536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf300f63f0 0x7fbf30106340 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fbf18009830 tx=0x7fbf18009b30).stop 2021-11-19T08:42:59.062 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:59.076+0000 7fbf3668f700 1 -- 172.21.15.23:0/3001791536 shutdown_connections 2021-11-19T08:42:59.062 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:59.076+0000 7fbf3668f700 1 --2- 172.21.15.23:0/3001791536 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf300f4a00 0x7fbf30105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:59.062 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:59.076+0000 7fbf3668f700 1 --2- 172.21.15.23:0/3001791536 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbf1c05c2a0 0x7fbf1c05e760 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:59.062 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:59.076+0000 7fbf3668f700 1 --2- 172.21.15.23:0/3001791536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf300f63f0 0x7fbf30106340 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:42:59.063 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:59.076+0000 7fbf3668f700 1 -- 172.21.15.23:0/3001791536 >> 172.21.15.23:0/3001791536 conn(0x7fbf300f04a0 msgr2=0x7fbf300fa320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:42:59.063 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:59.076+0000 7fbf3668f700 1 -- 172.21.15.23:0/3001791536 shutdown_connections 2021-11-19T08:42:59.063 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:42:59.076+0000 7fbf3668f700 1 -- 172.21.15.23:0/3001791536 wait complete. 2021-11-19T08:42:59.063 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 54 2021-11-19T08:42:59.073 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:42:48.445915+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24373, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 46, 'state': 'up:rejoin', 'state_seq': 23, 'addr': '172.21.15.134:6825/4111632986', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 4111632986}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 4111632986}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:42:59.073 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:42:59.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:59 smithi134 conmon[36363]: audit 2021-11-19T08:42:59.075128+0000 mon.smithi023 (mon.0) 684 : 2021-11-19T08:42:59.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:42:59 smithi134 conmon[36363]: audit [DBG] from='client.? 172.21.15.23:0/3001791536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:42:59.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:42:59 smithi023 conmon[33490]: audit 2021-11-19T08:42:59.075128+0000 mon.smithi023 (mon.0) 684 : audit [DBG] from='client.? 172.21.15.23:0/3001791536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:00.074 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:00.208 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.222+0000 7f03b6a1c700 1 -- 172.21.15.23:0/2514066290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b00f6820 msgr2=0x7f03b00f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:00.209 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.222+0000 7f03b6a1c700 1 --2- 172.21.15.23:0/2514066290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b00f6820 0x7f03b00f6c80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f03a0004660 tx=0x7f03a0009b30).stop 2021-11-19T08:43:00.209 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.222+0000 7f03b6a1c700 1 -- 172.21.15.23:0/2514066290 shutdown_connections 2021-11-19T08:43:00.209 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.222+0000 7f03b6a1c700 1 --2- 172.21.15.23:0/2514066290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b00f6820 0x7f03b00f6c80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:00.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.222+0000 7f03b6a1c700 1 --2- 172.21.15.23:0/2514066290 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03b00f5580 0x7f03b00f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:00.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.222+0000 7f03b6a1c700 1 -- 172.21.15.23:0/2514066290 >> 172.21.15.23:0/2514066290 conn(0x7f03b00f0ce0 msgr2=0x7f03b00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:00.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.222+0000 7f03b6a1c700 1 -- 172.21.15.23:0/2514066290 shutdown_connections 2021-11-19T08:43:00.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.222+0000 7f03b6a1c700 1 -- 172.21.15.23:0/2514066290 wait complete. 2021-11-19T08:43:00.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.222+0000 7f03b6a1c700 1 Processor -- start 2021-11-19T08:43:00.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.222+0000 7f03b6a1c700 1 -- start start 2021-11-19T08:43:00.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03b6a1c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b00f5580 0x7f03b0060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:00.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03b6a1c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03b00f6820 0x7f03b010bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:00.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03b6a1c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b010c1a0 con 0x7f03b00f5580 2021-11-19T08:43:00.212 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03b6a1c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b010c2e0 con 0x7f03b00f6820 2021-11-19T08:43:00.212 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03affff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b00f5580 0x7f03b0060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:00.212 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03affff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b00f5580 0x7f03b0060510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47638/0 (socket says 172.21.15.23:47638) 2021-11-19T08:43:00.212 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03affff700 1 -- 172.21.15.23:0/850531869 learned_addr learned my addr 172.21.15.23:0/850531869 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:00.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03af7fe700 1 --2- 172.21.15.23:0/850531869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03b00f6820 0x7f03b010bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:00.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03affff700 1 -- 172.21.15.23:0/850531869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03b00f6820 msgr2=0x7f03b010bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:00.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03affff700 1 --2- 172.21.15.23:0/850531869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03b00f6820 0x7f03b010bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:00.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03affff700 1 -- 172.21.15.23:0/850531869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a0005040 con 0x7f03b00f5580 2021-11-19T08:43:00.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03affff700 1 --2- 172.21.15.23:0/850531869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b00f5580 0x7f03b0060510 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f0398002730 tx=0x7f0398009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:00.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03acff9700 1 -- 172.21.15.23:0/850531869 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0398009320 con 0x7f03b00f5580 2021-11-19T08:43:00.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03b6a1c700 1 -- 172.21.15.23:0/850531869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b010c540 con 0x7f03b00f5580 2021-11-19T08:43:00.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03acff9700 1 -- 172.21.15.23:0/850531869 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f039800f040 con 0x7f03b00f5580 2021-11-19T08:43:00.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03acff9700 1 -- 172.21.15.23:0/850531869 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0398014840 con 0x7f03b00f5580 2021-11-19T08:43:00.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.223+0000 7f03b6a1c700 1 -- 172.21.15.23:0/850531869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b010cb00 con 0x7f03b00f5580 2021-11-19T08:43:00.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.224+0000 7f03b6a1c700 1 -- 172.21.15.23:0/850531869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b0027a00 con 0x7f03b00f5580 2021-11-19T08:43:00.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.225+0000 7f03acff9700 1 -- 172.21.15.23:0/850531869 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0398009480 con 0x7f03b00f5580 2021-11-19T08:43:00.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.225+0000 7f03acff9700 1 --2- 172.21.15.23:0/850531869 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f039c05c2a0 0x7f039c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:00.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.225+0000 7f03acff9700 1 -- 172.21.15.23:0/850531869 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f0398076510 con 0x7f03b00f5580 2021-11-19T08:43:00.217 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.225+0000 7f03af7fe700 1 --2- 172.21.15.23:0/850531869 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f039c05c2a0 0x7f039c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:00.217 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.226+0000 7f03af7fe700 1 --2- 172.21.15.23:0/850531869 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f039c05c2a0 0x7f039c05e760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f03a0004e30 tx=0x7f03a0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:00.217 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.229+0000 7f03acff9700 1 -- 172.21.15.23:0/850531869 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0398048ec0 con 0x7f03b00f5580 2021-11-19T08:43:00.407 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.423+0000 7f03b6a1c700 1 -- 172.21.15.23:0/850531869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03b00fdd10 con 0x7f03b00f5580 2021-11-19T08:43:00.408 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.424+0000 7f03acff9700 1 -- 172.21.15.23:0/850531869 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f039804c4e0 con 0x7f03b00f5580 2021-11-19T08:43:00.409 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:00.409 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":53,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:42:48.445915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24373,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":46,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.134:6825/4111632986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":4111632986},{"type":"v1","addr":"172.21.15.134:6825","nonce":4111632986}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:00.411 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.425+0000 7f03b6a1c700 1 -- 172.21.15.23:0/850531869 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f039c05c2a0 msgr2=0x7f039c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:00.411 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.425+0000 7f03b6a1c700 1 --2- 172.21.15.23:0/850531869 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f039c05c2a0 0x7f039c05e760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f03a0004e30 tx=0x7f03a0009b30).stop 2021-11-19T08:43:00.412 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.425+0000 7f03b6a1c700 1 -- 172.21.15.23:0/850531869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b00f5580 msgr2=0x7f03b0060510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:00.412 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.425+0000 7f03b6a1c700 1 --2- 172.21.15.23:0/850531869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b00f5580 0x7f03b0060510 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f0398002730 tx=0x7f0398009d80).stop 2021-11-19T08:43:00.412 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.425+0000 7f03b6a1c700 1 -- 172.21.15.23:0/850531869 shutdown_connections 2021-11-19T08:43:00.412 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.425+0000 7f03b6a1c700 1 --2- 172.21.15.23:0/850531869 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f039c05c2a0 0x7f039c05e760 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:00.413 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.425+0000 7f03b6a1c700 1 --2- 172.21.15.23:0/850531869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b00f5580 0x7f03b0060510 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:00.413 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.425+0000 7f03b6a1c700 1 --2- 172.21.15.23:0/850531869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03b00f6820 0x7f03b010bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:00.413 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.425+0000 7f03b6a1c700 1 -- 172.21.15.23:0/850531869 >> 172.21.15.23:0/850531869 conn(0x7f03b00f0ce0 msgr2=0x7f03b00fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:00.413 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.426+0000 7f03b6a1c700 1 -- 172.21.15.23:0/850531869 shutdown_connections 2021-11-19T08:43:00.414 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:00.426+0000 7f03b6a1c700 1 -- 172.21.15.23:0/850531869 wait complete. 2021-11-19T08:43:00.414 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 54 2021-11-19T08:43:00.422 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:42:48.445915+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24373, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 46, 'state': 'up:rejoin', 'state_seq': 23, 'addr': '172.21.15.134:6825/4111632986', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 4111632986}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 4111632986}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:00.423 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:43:00.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:00 smithi134 conmon[36363]: cluster 2021-11-19T08:42:58.867454+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:43:00.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:00 smithi134 conmon[36363]: mgr.14164) 269 : cluster [DBG] pgmap v238: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2021-11-19T08:43:00.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:00 smithi134 conmon[36363]: audit 2021-11-19T08:43:00.424720+0000 mon.smithi023 (mon.0) 685 : audit [DBG] 2021-11-19T08:43:00.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:00 smithi134 conmon[36363]: from='client.? 172.21.15.23:0/850531869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:00.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:00 smithi023 conmon[33490]: cluster 2021-11-19T08:42:58.867454+0000 mgr.smithi023.ehlpcl (mgr.14164) 269 : cluster 2021-11-19T08:43:00.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:00 smithi023 conmon[33490]: [DBG] pgmap v238: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2021-11-19T08:43:00.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:00 smithi023 conmon[33490]: audit 2021-11-19T08:43:00.424720+0000 mon.smithi023 (mon.0) 685 : audit [DBG] 2021-11-19T08:43:00.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:00 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/850531869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:01.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:01.560 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.573+0000 7f0f83a06700 1 -- 172.21.15.23:0/2361994217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f7c0f55a0 msgr2=0x7f0f7c0f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:01.560 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.573+0000 7f0f83a06700 1 --2- 172.21.15.23:0/2361994217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f7c0f55a0 0x7f0f7c0f59c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f0f740099e0 tx=0x7f0f74004d10).stop 2021-11-19T08:43:01.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.573+0000 7f0f83a06700 1 -- 172.21.15.23:0/2361994217 shutdown_connections 2021-11-19T08:43:01.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.573+0000 7f0f83a06700 1 --2- 172.21.15.23:0/2361994217 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f7c0f6840 0x7f0f7c0f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:01.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.573+0000 7f0f83a06700 1 --2- 172.21.15.23:0/2361994217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f7c0f55a0 0x7f0f7c0f59c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:01.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.573+0000 7f0f83a06700 1 -- 172.21.15.23:0/2361994217 >> 172.21.15.23:0/2361994217 conn(0x7f0f7c0f0ce0 msgr2=0x7f0f7c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:01.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.573+0000 7f0f83a06700 1 -- 172.21.15.23:0/2361994217 shutdown_connections 2021-11-19T08:43:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.573+0000 7f0f83a06700 1 -- 172.21.15.23:0/2361994217 wait complete. 2021-11-19T08:43:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.574+0000 7f0f83a06700 1 Processor -- start 2021-11-19T08:43:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.574+0000 7f0f83a06700 1 -- start start 2021-11-19T08:43:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.574+0000 7f0f83a06700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f7c0f55a0 0x7f0f7c106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:01.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.574+0000 7f0f83a06700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f7c0f6840 0x7f0f7c1074a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:01.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.574+0000 7f0f83a06700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c107a50 con 0x7f0f7c0f55a0 2021-11-19T08:43:01.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.574+0000 7f0f83a06700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c107b90 con 0x7f0f7c0f6840 2021-11-19T08:43:01.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.574+0000 7f0f817a2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f7c0f55a0 0x7f0f7c106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:01.564 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.574+0000 7f0f817a2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f7c0f55a0 0x7f0f7c106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47652/0 (socket says 172.21.15.23:47652) 2021-11-19T08:43:01.564 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.574+0000 7f0f817a2700 1 -- 172.21.15.23:0/378618626 learned_addr learned my addr 172.21.15.23:0/378618626 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:01.564 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.574+0000 7f0f80fa1700 1 --2- 172.21.15.23:0/378618626 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f7c0f6840 0x7f0f7c1074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:01.564 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.574+0000 7f0f817a2700 1 -- 172.21.15.23:0/378618626 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f7c0f6840 msgr2=0x7f0f7c1074a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.574+0000 7f0f817a2700 1 --2- 172.21.15.23:0/378618626 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f7c0f6840 0x7f0f7c1074a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.574+0000 7f0f817a2700 1 -- 172.21.15.23:0/378618626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f74009640 con 0x7f0f7c0f55a0 2021-11-19T08:43:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.574+0000 7f0f817a2700 1 --2- 172.21.15.23:0/378618626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f7c0f55a0 0x7f0f7c106fb0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f0f740078d0 tx=0x7f0f74004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.575+0000 7f0f79ffb700 1 -- 172.21.15.23:0/378618626 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f7400e070 con 0x7f0f7c0f55a0 2021-11-19T08:43:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.575+0000 7f0f79ffb700 1 -- 172.21.15.23:0/378618626 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f74007d70 con 0x7f0f7c0f55a0 2021-11-19T08:43:01.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.575+0000 7f0f83a06700 1 -- 172.21.15.23:0/378618626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f7c10c5d0 con 0x7f0f7c0f55a0 2021-11-19T08:43:01.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.575+0000 7f0f79ffb700 1 -- 172.21.15.23:0/378618626 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f74018070 con 0x7f0f7c0f55a0 2021-11-19T08:43:01.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.575+0000 7f0f83a06700 1 -- 172.21.15.23:0/378618626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7c10cb90 con 0x7f0f7c0f55a0 2021-11-19T08:43:01.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.576+0000 7f0f83a06700 1 -- 172.21.15.23:0/378618626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7c0fdd30 con 0x7f0f7c0f55a0 2021-11-19T08:43:01.567 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.576+0000 7f0f79ffb700 1 -- 172.21.15.23:0/378618626 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0f74024060 con 0x7f0f7c0f55a0 2021-11-19T08:43:01.568 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.576+0000 7f0f79ffb700 1 --2- 172.21.15.23:0/378618626 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f6405c2a0 0x7f0f6405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:01.568 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.577+0000 7f0f79ffb700 1 -- 172.21.15.23:0/378618626 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f0f74077840 con 0x7f0f7c0f55a0 2021-11-19T08:43:01.568 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.577+0000 7f0f80fa1700 1 --2- 172.21.15.23:0/378618626 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f6405c2a0 0x7f0f6405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:01.569 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.577+0000 7f0f80fa1700 1 --2- 172.21.15.23:0/378618626 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f6405c2a0 0x7f0f6405e760 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f0f70009fd0 tx=0x7f0f7000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:01.569 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.580+0000 7f0f79ffb700 1 -- 172.21.15.23:0/378618626 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f7404a210 con 0x7f0f7c0f55a0 2021-11-19T08:43:01.763 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.778+0000 7f0f83a06700 1 -- 172.21.15.23:0/378618626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f7c10d2e0 con 0x7f0f7c0f55a0 2021-11-19T08:43:01.763 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.778+0000 7f0f79ffb700 1 -- 172.21.15.23:0/378618626 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f0f7404d830 con 0x7f0f7c0f55a0 2021-11-19T08:43:01.764 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:01.764 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":53,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:42:48.445915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24373,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":46,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.134:6825/4111632986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":4111632986},{"type":"v1","addr":"172.21.15.134:6825","nonce":4111632986}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:01.765 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.780+0000 7f0f83a06700 1 -- 172.21.15.23:0/378618626 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f6405c2a0 msgr2=0x7f0f6405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:01.765 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.780+0000 7f0f83a06700 1 --2- 172.21.15.23:0/378618626 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f6405c2a0 0x7f0f6405e760 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f0f70009fd0 tx=0x7f0f7000b040).stop 2021-11-19T08:43:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.780+0000 7f0f83a06700 1 -- 172.21.15.23:0/378618626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f7c0f55a0 msgr2=0x7f0f7c106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.780+0000 7f0f83a06700 1 --2- 172.21.15.23:0/378618626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f7c0f55a0 0x7f0f7c106fb0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f0f740078d0 tx=0x7f0f74004d10).stop 2021-11-19T08:43:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.780+0000 7f0f83a06700 1 -- 172.21.15.23:0/378618626 shutdown_connections 2021-11-19T08:43:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.780+0000 7f0f83a06700 1 --2- 172.21.15.23:0/378618626 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f6405c2a0 0x7f0f6405e760 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:01.767 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.780+0000 7f0f83a06700 1 --2- 172.21.15.23:0/378618626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f7c0f55a0 0x7f0f7c106fb0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:01.767 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.780+0000 7f0f83a06700 1 --2- 172.21.15.23:0/378618626 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f7c0f6840 0x7f0f7c1074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:01.767 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.780+0000 7f0f83a06700 1 -- 172.21.15.23:0/378618626 >> 172.21.15.23:0/378618626 conn(0x7f0f7c0f0ce0 msgr2=0x7f0f7c05fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:01.767 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.780+0000 7f0f83a06700 1 -- 172.21.15.23:0/378618626 shutdown_connections 2021-11-19T08:43:01.767 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:01.780+0000 7f0f83a06700 1 -- 172.21.15.23:0/378618626 wait complete. 2021-11-19T08:43:01.768 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 54 2021-11-19T08:43:01.777 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:42:48.445915+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24373, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 46, 'state': 'up:rejoin', 'state_seq': 23, 'addr': '172.21.15.134:6825/4111632986', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 4111632986}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 4111632986}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:01.777 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:43:02.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:02.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:02 smithi134 conmon[36363]: cluster 2021-11-19T08:43:00.868005+0000 mgr.smithi023.ehlpcl (mgr.14164) 270 : cluster [DBG] pgmap v239: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2021-11-19T08:43:02.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:02 smithi134 conmon[36363]: audit 2021-11-19T08:43:01.779307+0000 mon.smithi023 (mon.0) 686 : audit [DBG] from='client.? 172.21.15.23:0/378618626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:02.808 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:02 smithi023 conmon[33490]: cluster 2021-11-19T08:43:00.868005+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:43:02.808 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:02 smithi023 conmon[33490]: .14164) 270 : cluster [DBG] pgmap v239: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2021-11-19T08:43:02.809 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:02 smithi023 conmon[33490]: audit 2021-11-19T08:43:01.779307+0000 mon.smithi023 (mon.0) 686 : audit [DBG] from='client.? 172.21.15.23:0/378618626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:02.914 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.928+0000 7f3056c28700 1 -- 172.21.15.23:0/1768656466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30500f6670 msgr2=0x7f30500f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:02.915 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.928+0000 7f3056c28700 1 --2- 172.21.15.23:0/1768656466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30500f6670 0x7f30500f6a90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f3044004660 tx=0x7f3044009b30).stop 2021-11-19T08:43:02.915 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.928+0000 7f3056c28700 1 -- 172.21.15.23:0/1768656466 shutdown_connections 2021-11-19T08:43:02.915 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.928+0000 7f3056c28700 1 --2- 172.21.15.23:0/1768656466 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30500f7910 0x7f30500f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:02.915 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.928+0000 7f3056c28700 1 --2- 172.21.15.23:0/1768656466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30500f6670 0x7f30500f6a90 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:02.916 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.928+0000 7f3056c28700 1 -- 172.21.15.23:0/1768656466 >> 172.21.15.23:0/1768656466 conn(0x7f30500f1db0 msgr2=0x7f30500f41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:02.916 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.928+0000 7f3056c28700 1 -- 172.21.15.23:0/1768656466 shutdown_connections 2021-11-19T08:43:02.916 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.928+0000 7f3056c28700 1 -- 172.21.15.23:0/1768656466 wait complete. 2021-11-19T08:43:02.916 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.928+0000 7f3056c28700 1 Processor -- start 2021-11-19T08:43:02.917 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.928+0000 7f3056c28700 1 -- start start 2021-11-19T08:43:02.917 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f3056c28700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30500f6670 0x7f305006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:02.917 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f3056c28700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30500f7910 0x7f3050069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:02.917 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f3056c28700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305006bd70 con 0x7f30500f7910 2021-11-19T08:43:02.917 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f3056c28700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305006a310 con 0x7f30500f6670 2021-11-19T08:43:02.918 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f304ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30500f7910 0x7f3050069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:02.918 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f304ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30500f7910 0x7f3050069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47662/0 (socket says 172.21.15.23:47662) 2021-11-19T08:43:02.918 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f304ffff700 1 -- 172.21.15.23:0/1272640767 learned_addr learned my addr 172.21.15.23:0/1272640767 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:02.918 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f30549c4700 1 --2- 172.21.15.23:0/1272640767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30500f6670 0x7f305006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:02.919 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f304ffff700 1 -- 172.21.15.23:0/1272640767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30500f6670 msgr2=0x7f305006b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:02.919 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f304ffff700 1 --2- 172.21.15.23:0/1272640767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30500f6670 0x7f305006b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:02.919 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f304ffff700 1 -- 172.21.15.23:0/1272640767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3044005040 con 0x7f30500f7910 2021-11-19T08:43:02.919 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f304ffff700 1 --2- 172.21.15.23:0/1272640767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30500f7910 0x7f3050069e20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f303c00c530 tx=0x7f303c007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:02.920 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f304d7fa700 1 -- 172.21.15.23:0/1272640767 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f303c00f650 con 0x7f30500f7910 2021-11-19T08:43:02.920 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f3056c28700 1 -- 172.21.15.23:0/1272640767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305006a590 con 0x7f30500f7910 2021-11-19T08:43:02.920 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f304d7fa700 1 -- 172.21.15.23:0/1272640767 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f303c00fc60 con 0x7f30500f7910 2021-11-19T08:43:02.920 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f304d7fa700 1 -- 172.21.15.23:0/1272640767 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f303c012c80 con 0x7f30500f7910 2021-11-19T08:43:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.929+0000 7f3056c28700 1 -- 172.21.15.23:0/1272640767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305006ab50 con 0x7f30500f7910 2021-11-19T08:43:02.922 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.930+0000 7f3056c28700 1 -- 172.21.15.23:0/1272640767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3050046330 con 0x7f30500f7910 2021-11-19T08:43:02.922 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.931+0000 7f304d7fa700 1 -- 172.21.15.23:0/1272640767 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f303c010070 con 0x7f30500f7910 2021-11-19T08:43:02.923 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.932+0000 7f304d7fa700 1 --2- 172.21.15.23:0/1272640767 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f304005c2a0 0x7f304005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:02.923 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.932+0000 7f304d7fa700 1 -- 172.21.15.23:0/1272640767 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6070+0+0 (secure 0 0 0) 0x7f303c076790 con 0x7f30500f7910 2021-11-19T08:43:02.923 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.932+0000 7f30549c4700 1 --2- 172.21.15.23:0/1272640767 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f304005c2a0 0x7f304005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:02.923 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.932+0000 7f30549c4700 1 --2- 172.21.15.23:0/1272640767 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f304005c2a0 0x7f304005e760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f3044004e30 tx=0x7f3044009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:02.924 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:02.935+0000 7f304d7fa700 1 -- 172.21.15.23:0/1272640767 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f303c04cbd0 con 0x7f30500f7910 2021-11-19T08:43:03.115 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:03.131+0000 7f3056c28700 1 -- 172.21.15.23:0/1272640767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30500fee00 con 0x7f30500f7910 2021-11-19T08:43:03.116 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:03.131+0000 7f304d7fa700 1 -- 172.21.15.23:0/1272640767 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f303c020090 con 0x7f30500f7910 2021-11-19T08:43:03.116 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:03.117 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":54},{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":53,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:42:48.445915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24373,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24373":{"gid":24373,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":46,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.134:6825/4111632986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":4111632986},{"type":"v1","addr":"172.21.15.134:6825","nonce":4111632986}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:03.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:03.133+0000 7f3056c28700 1 -- 172.21.15.23:0/1272640767 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f304005c2a0 msgr2=0x7f304005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:03.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:03.133+0000 7f3056c28700 1 --2- 172.21.15.23:0/1272640767 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f304005c2a0 0x7f304005e760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f3044004e30 tx=0x7f3044009b30).stop 2021-11-19T08:43:03.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:03.133+0000 7f3056c28700 1 -- 172.21.15.23:0/1272640767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30500f7910 msgr2=0x7f3050069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:03.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:03.133+0000 7f3056c28700 1 --2- 172.21.15.23:0/1272640767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30500f7910 0x7f3050069e20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f303c00c530 tx=0x7f303c007e20).stop 2021-11-19T08:43:03.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:03.133+0000 7f3056c28700 1 -- 172.21.15.23:0/1272640767 shutdown_connections 2021-11-19T08:43:03.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:03.133+0000 7f3056c28700 1 --2- 172.21.15.23:0/1272640767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30500f6670 0x7f305006b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:03.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:03.133+0000 7f3056c28700 1 --2- 172.21.15.23:0/1272640767 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f304005c2a0 0x7f304005e760 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:03.120 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:03.133+0000 7f3056c28700 1 --2- 172.21.15.23:0/1272640767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30500f7910 0x7f3050069e20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:03.120 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:03.133+0000 7f3056c28700 1 -- 172.21.15.23:0/1272640767 >> 172.21.15.23:0/1272640767 conn(0x7f30500f1db0 msgr2=0x7f30500fb360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:03.120 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:03.133+0000 7f3056c28700 1 -- 172.21.15.23:0/1272640767 shutdown_connections 2021-11-19T08:43:03.120 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:03.133+0000 7f3056c28700 1 -- 172.21.15.23:0/1272640767 wait complete. 2021-11-19T08:43:03.121 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 54 2021-11-19T08:43:03.129 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:42:48.445915+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 49, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24373, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24373': {'gid': 24373, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 46, 'state': 'up:rejoin', 'state_seq': 23, 'addr': '172.21.15.134:6825/4111632986', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 4111632986}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 4111632986}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:03.130 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:43:03.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:03 smithi134 conmon[36363]: audit 2021-11-19T08:43:03.132258+0000 mon.smithi023 (mon.0) 687 : audit [DBG] from='client.? 172.21.15.23:0/1272640767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:03.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:03 smithi134 conmon[36363]: cluster 2021-11-19T08:43:03.348101+0000 mon.smithi023 2021-11-19T08:43:03.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:03 smithi134 conmon[36363]: (mon.0) 688 : cluster [INF] daemon mds.cephfs.smithi134.jqsnzo restarted 2021-11-19T08:43:03.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:03 smithi134 conmon[36363]: cluster 2021-11-19T08:43:03.356533+0000 mon.smithi023 (mon.0) 689 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2021-11-19T08:43:03.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:03 smithi023 conmon[33490]: audit 2021-11-19T08:43:03.132258+0000 mon.smithi023 2021-11-19T08:43:03.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:03 smithi023 conmon[33490]: (mon.0) 687 : audit [DBG] from='client.? 172.21.15.23:0/1272640767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:03.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:03 smithi023 conmon[33490]: cluster 2021-11-19T08:43:03.348101+0000 mon.smithi023 (mon.0) 688 : cluster 2021-11-19T08:43:03.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:03 smithi023 conmon[33490]: [INF] daemon mds.cephfs.smithi134.jqsnzo restarted 2021-11-19T08:43:03.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:03 smithi023 conmon[33490]: cluster 2021-11-19T08:43 2021-11-19T08:43:03.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:03 smithi023 conmon[33490]: :03.356533+0000 mon.smithi023 (mon.0) 689 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2021-11-19T08:43:04.131 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:04.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.279+0000 7ff657981700 1 -- 172.21.15.23:0/3412269859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6500fbb70 msgr2=0x7ff6500fdf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:04.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.279+0000 7ff657981700 1 --2- 172.21.15.23:0/3412269859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6500fbb70 0x7ff6500fdf60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7ff644004660 tx=0x7ff644009b30).stop 2021-11-19T08:43:04.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.279+0000 7ff657981700 1 -- 172.21.15.23:0/3412269859 shutdown_connections 2021-11-19T08:43:04.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.279+0000 7ff657981700 1 --2- 172.21.15.23:0/3412269859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6500fbb70 0x7ff6500fdf60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:04.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.279+0000 7ff657981700 1 --2- 172.21.15.23:0/3412269859 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff650060140 0x7ff6500fb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:04.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.279+0000 7ff657981700 1 -- 172.21.15.23:0/3412269859 >> 172.21.15.23:0/3412269859 conn(0x7ff6500f0ce0 msgr2=0x7ff6500f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:04.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.279+0000 7ff657981700 1 -- 172.21.15.23:0/3412269859 shutdown_connections 2021-11-19T08:43:04.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.279+0000 7ff657981700 1 -- 172.21.15.23:0/3412269859 wait complete. 2021-11-19T08:43:04.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.279+0000 7ff657981700 1 Processor -- start 2021-11-19T08:43:04.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff657981700 1 -- start start 2021-11-19T08:43:04.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff657981700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff650060140 0x7ff6501090c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:04.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff657981700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6500fbb70 0x7ff6501095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:04.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff657981700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff650109b60 con 0x7ff650060140 2021-11-19T08:43:04.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff657981700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff650109ca0 con 0x7ff6500fbb70 2021-11-19T08:43:04.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff65571d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff650060140 0x7ff6501090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:04.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff65571d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff650060140 0x7ff6501090c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47680/0 (socket says 172.21.15.23:47680) 2021-11-19T08:43:04.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff65571d700 1 -- 172.21.15.23:0/437387259 learned_addr learned my addr 172.21.15.23:0/437387259 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:04.270 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff654f1c700 1 --2- 172.21.15.23:0/437387259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6500fbb70 0x7ff6501095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:04.270 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff65571d700 1 -- 172.21.15.23:0/437387259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6500fbb70 msgr2=0x7ff6501095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:04.270 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff65571d700 1 --2- 172.21.15.23:0/437387259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6500fbb70 0x7ff6501095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:04.270 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff65571d700 1 -- 172.21.15.23:0/437387259 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff644005040 con 0x7ff650060140 2021-11-19T08:43:04.271 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff654f1c700 1 --2- 172.21.15.23:0/437387259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6500fbb70 0x7ff6501095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:43:04.271 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff65571d700 1 --2- 172.21.15.23:0/437387259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff650060140 0x7ff6501090c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7ff64800c790 tx=0x7ff648007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:04.271 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff64dffb700 1 -- 172.21.15.23:0/437387259 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff64800e680 con 0x7ff650060140 2021-11-19T08:43:04.271 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff64dffb700 1 -- 172.21.15.23:0/437387259 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff64800ec90 con 0x7ff650060140 2021-11-19T08:43:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff64dffb700 1 -- 172.21.15.23:0/437387259 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff648013d60 con 0x7ff650060140 2021-11-19T08:43:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.280+0000 7ff657981700 1 -- 172.21.15.23:0/437387259 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65010e740 con 0x7ff650060140 2021-11-19T08:43:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.281+0000 7ff657981700 1 -- 172.21.15.23:0/437387259 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65010ed00 con 0x7ff650060140 2021-11-19T08:43:04.273 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.282+0000 7ff657981700 1 -- 172.21.15.23:0/437387259 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff650027a00 con 0x7ff650060140 2021-11-19T08:43:04.273 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.282+0000 7ff64dffb700 1 -- 172.21.15.23:0/437387259 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff64801a060 con 0x7ff650060140 2021-11-19T08:43:04.274 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.282+0000 7ff64dffb700 1 --2- 172.21.15.23:0/437387259 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff63805c2a0 0x7ff63805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:04.274 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.282+0000 7ff64dffb700 1 -- 172.21.15.23:0/437387259 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7ff648076360 con 0x7ff650060140 2021-11-19T08:43:04.274 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.282+0000 7ff654f1c700 1 --2- 172.21.15.23:0/437387259 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff63805c2a0 0x7ff63805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:04.274 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.283+0000 7ff654f1c700 1 --2- 172.21.15.23:0/437387259 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff63805c2a0 0x7ff63805e760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff644009d60 tx=0x7ff644009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:04.275 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.286+0000 7ff64dffb700 1 -- 172.21.15.23:0/437387259 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff648048da0 con 0x7ff650060140 2021-11-19T08:43:04.476 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.492+0000 7ff657981700 1 -- 172.21.15.23:0/437387259 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff650101040 con 0x7ff650060140 2021-11-19T08:43:04.477 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.493+0000 7ff64dffb700 1 -- 172.21.15.23:0/437387259 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2948 (secure 0 0 0) 0x7ff64804c3c0 con 0x7ff650060140 2021-11-19T08:43:04.478 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:04.478 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44},{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":55}],"filesystems":[{"mdsmap":{"epoch":56,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:03.528657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:replay","state_seq":2,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:04.480 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.495+0000 7ff657981700 1 -- 172.21.15.23:0/437387259 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff63805c2a0 msgr2=0x7ff63805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:04.480 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.495+0000 7ff657981700 1 --2- 172.21.15.23:0/437387259 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff63805c2a0 0x7ff63805e760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff644009d60 tx=0x7ff644009b30).stop 2021-11-19T08:43:04.481 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.495+0000 7ff657981700 1 -- 172.21.15.23:0/437387259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff650060140 msgr2=0x7ff6501090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:04.481 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.495+0000 7ff657981700 1 --2- 172.21.15.23:0/437387259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff650060140 0x7ff6501090c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7ff64800c790 tx=0x7ff648007d70).stop 2021-11-19T08:43:04.482 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.495+0000 7ff657981700 1 -- 172.21.15.23:0/437387259 shutdown_connections 2021-11-19T08:43:04.482 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.495+0000 7ff657981700 1 --2- 172.21.15.23:0/437387259 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff63805c2a0 0x7ff63805e760 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:04.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.495+0000 7ff657981700 1 --2- 172.21.15.23:0/437387259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff650060140 0x7ff6501090c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:04.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.495+0000 7ff657981700 1 --2- 172.21.15.23:0/437387259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6500fbb70 0x7ff6501095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:04.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.495+0000 7ff657981700 1 -- 172.21.15.23:0/437387259 >> 172.21.15.23:0/437387259 conn(0x7ff6500f0ce0 msgr2=0x7ff6500f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:04.484 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.495+0000 7ff657981700 1 -- 172.21.15.23:0/437387259 shutdown_connections 2021-11-19T08:43:04.484 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:04.495+0000 7ff657981700 1 -- 172.21.15.23:0/437387259 wait complete. 2021-11-19T08:43:04.485 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 56 2021-11-19T08:43:04.492 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:03.528657+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:04.493 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:43:04.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: cluster 2021-11-19T08:43:02.868404+0000 mgr.smithi023.ehlpcl (mgr.14164) 271 : cluster [DBG] pgmap v240: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:43:04.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: 2021-11-19T08:43:04.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: cluster 2021-11-19T08:43:03.528552+0000 mon.smithi023 (mon.0) 690 : cluster [DBG] mds.? [v2:172.21.15.134:6824/464801254,v1:172.21.15.134:6825/464801254] up:boot 2021-11-19T08:43:04.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: cluster 2021-11-19T08:43:03.528618+0000 2021-11-19T08:43:04.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: mon.smithi023 (mon.0) 691 : cluster [INF] Standby daemon mds.cephfs.smithi023.fuvzkr assigned to filesystem cephfs as rank 0 2021-11-19T08:43:04.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: cluster 2021-11- 2021-11-19T08:43:04.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: 19T08:43:03.528940+0000 mon.smithi023 (mon.0) 692 : cluster 2021-11-19T08:43:04.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: [DBG] fsmap cephfs:1/2 {1=cephfs.smithi023.wazanl=up:rejoin} 3 up:standby, 1 failed 2021-11-19T08:43:04.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: audit 2021-11-19T08:43: 2021-11-19T08:43:04.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: 03.529042+0000 mon.smithi023 (mon.0) 693 : audit 2021-11-19T08:43:04.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.jqsnzo"}]: dispatch 2021-11-19T08:43:04.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: cluster 2021-11-19T08:43 2021-11-19T08:43:04.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: :03.535617+0000 mon.smithi023 (mon.0) 694 2021-11-19T08:43:04.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:replay,1=cephfs.smithi023.wazanl=up:rejoin} 2 up:standby 2021-11-19T08:43:04.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: audit 2021-11-19T08: 2021-11-19T08:43:04.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: 43:04.494208+0000 mon.smithi023 (mon.0) 2021-11-19T08:43:04.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:04 smithi134 conmon[36363]: 695 : audit [DBG] from='client.? 172.21.15.23:0/437387259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:04.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: cluster 2021-11-19T08:43:02.868404+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:43:04.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: mgr.14164) 271 : cluster [DBG] pgmap v240: 65 pgs: 65 active+clean; 4.7 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:43:04.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: cluster 2021-11-19T08:43:03.528552 2021-11-19T08:43:04.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 690 : 2021-11-19T08:43:04.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: cluster [DBG] mds.? [v2:172.21.15.134:6824/464801254,v1:172.21.15.134:6825/464801254] up:boot 2021-11-19T08:43:04.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: cluster 2021- 2021-11-19T08:43:04.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: 11-19T08:43:03.528618+0000 mon.smithi023 ( 2021-11-19T08:43:04.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: mon.0) 691 : cluster [INF] Standby daemon mds.cephfs.smithi023.fuvzkr assigned to filesystem cephfs as rank 0 2021-11-19T08:43:04.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: cluster 2021-11-19T08:43:04.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: 2021-11-19T08:43:03.528940+0000 2021-11-19T08:43:04.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: mon.smithi023 (mon.0) 692 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi023.wazanl=up:rejoin} 3 up:standby, 1 failed 2021-11-19T08:43:04.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: 2021-11-19T08:43:04.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: audit 2021-11-19T08:43: 2021-11-19T08:43:04.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: 03.529042+0000 mon.smithi023 (mon.0) 693 2021-11-19T08:43:04.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.jqsnzo"}]: dispatch 2021-11-19T08:43:04.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:43:04.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: 08:43:03.535617+0000 mon.smithi023 (mon. 2021-11-19T08:43:04.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: 0) 694 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:replay,1=cephfs.smithi023.wazanl=up:rejoin} 2 up:standby 2021-11-19T08:43:04.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: audit 2021 2021-11-19T08:43:04.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: -11-19T08:43:04.494208+0000 2021-11-19T08:43:04.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: mon.smithi023 (mon.0) 695 : audit [DBG] from='client.? 172.21.15.23:0/437387259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:04.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:04 smithi023 conmon[33490]: 2021-11-19T08:43:05.495 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:05.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.648+0000 7f4ea0a64700 1 -- 172.21.15.23:0/2689444537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e9c0f6820 msgr2=0x7f4e9c0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:05.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.648+0000 7f4ea0a64700 1 --2- 172.21.15.23:0/2689444537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e9c0f6820 0x7f4e9c0f6c80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f4e90004660 tx=0x7f4e90009b30).stop 2021-11-19T08:43:05.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.648+0000 7f4ea0a64700 1 -- 172.21.15.23:0/2689444537 shutdown_connections 2021-11-19T08:43:05.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.648+0000 7f4ea0a64700 1 --2- 172.21.15.23:0/2689444537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e9c0f6820 0x7f4e9c0f6c80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:05.639 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.648+0000 7f4ea0a64700 1 --2- 172.21.15.23:0/2689444537 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e9c0f5580 0x7f4e9c0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:05.639 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.648+0000 7f4ea0a64700 1 -- 172.21.15.23:0/2689444537 >> 172.21.15.23:0/2689444537 conn(0x7f4e9c0f0ce0 msgr2=0x7f4e9c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:05.639 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.648+0000 7f4ea0a64700 1 -- 172.21.15.23:0/2689444537 shutdown_connections 2021-11-19T08:43:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.648+0000 7f4ea0a64700 1 -- 172.21.15.23:0/2689444537 wait complete. 2021-11-19T08:43:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4ea0a64700 1 Processor -- start 2021-11-19T08:43:05.642 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4ea0a64700 1 -- start start 2021-11-19T08:43:05.642 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4ea0a64700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e9c0f5580 0x7f4e9c0604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:05.642 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4ea0a64700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e9c0f6820 0x7f4e9c10bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:05.643 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4ea0a64700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c10c120 con 0x7f4e9c0f5580 2021-11-19T08:43:05.643 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4ea0a64700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c10c260 con 0x7f4e9c0f6820 2021-11-19T08:43:05.643 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4e9a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e9c0f5580 0x7f4e9c0604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:05.643 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4e9a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e9c0f5580 0x7f4e9c0604f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47700/0 (socket says 172.21.15.23:47700) 2021-11-19T08:43:05.644 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4e9a59c700 1 -- 172.21.15.23:0/1227237801 learned_addr learned my addr 172.21.15.23:0/1227237801 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:05.644 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4e99d9b700 1 --2- 172.21.15.23:0/1227237801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e9c0f6820 0x7f4e9c10bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:05.644 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4e9a59c700 1 -- 172.21.15.23:0/1227237801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e9c0f6820 msgr2=0x7f4e9c10bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:05.644 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4e9a59c700 1 --2- 172.21.15.23:0/1227237801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e9c0f6820 0x7f4e9c10bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:05.644 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4e9a59c700 1 -- 172.21.15.23:0/1227237801 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e90005040 con 0x7f4e9c0f5580 2021-11-19T08:43:05.645 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4e9a59c700 1 --2- 172.21.15.23:0/1227237801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e9c0f5580 0x7f4e9c0604f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f4e8c002700 tx=0x7f4e8c009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:05.646 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4e8affd700 1 -- 172.21.15.23:0/1227237801 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e8c009320 con 0x7f4e9c0f5580 2021-11-19T08:43:05.646 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.649+0000 7f4e8affd700 1 -- 172.21.15.23:0/1227237801 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e8c00f040 con 0x7f4e9c0f5580 2021-11-19T08:43:05.647 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.650+0000 7f4ea0a64700 1 -- 172.21.15.23:0/1227237801 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e9c10c4c0 con 0x7f4e9c0f5580 2021-11-19T08:43:05.647 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.650+0000 7f4e8affd700 1 -- 172.21.15.23:0/1227237801 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e8c0148e0 con 0x7f4e9c0f5580 2021-11-19T08:43:05.647 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.650+0000 7f4ea0a64700 1 -- 172.21.15.23:0/1227237801 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9c10ca50 con 0x7f4e9c0f5580 2021-11-19T08:43:05.647 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.650+0000 7f4ea0a64700 1 -- 172.21.15.23:0/1227237801 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e9c046330 con 0x7f4e9c0f5580 2021-11-19T08:43:05.647 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.651+0000 7f4e8affd700 1 -- 172.21.15.23:0/1227237801 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4e8c009480 con 0x7f4e9c0f5580 2021-11-19T08:43:05.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.651+0000 7f4e8affd700 1 --2- 172.21.15.23:0/1227237801 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4e8405c2a0 0x7f4e8405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:05.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.651+0000 7f4e8affd700 1 -- 172.21.15.23:0/1227237801 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f4e8c0763e0 con 0x7f4e9c0f5580 2021-11-19T08:43:05.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.651+0000 7f4e99d9b700 1 --2- 172.21.15.23:0/1227237801 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4e8405c2a0 0x7f4e8405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:05.649 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.652+0000 7f4e99d9b700 1 --2- 172.21.15.23:0/1227237801 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4e8405c2a0 0x7f4e8405e760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f4e90004e30 tx=0x7f4e90009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:05.649 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.655+0000 7f4e8affd700 1 -- 172.21.15.23:0/1227237801 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e8c048e20 con 0x7f4e9c0f5580 2021-11-19T08:43:05.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:05 smithi134 conmon[36363]: cluster 2021-11-19T08:43:04.540429+0000 mon.smithi023 (mon.0) 696 : cluster [DBG] mds.? [v2:172.21.15.23:6826/2135889102,v1:172.21.15.23:6827/2135889102] up:resolve 2021-11-19T08:43:05.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:05 smithi134 conmon[36363]: cluster 2021-11-19T08:43:04.540470+0000 mon.smithi023 (mon.0) 697 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi023.wazanl=up:rejoin} 2 up:standby 2021-11-19T08:43:05.834 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.849+0000 7f4ea0a64700 1 -- 172.21.15.23:0/1227237801 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e9c0fdd10 con 0x7f4e9c0f5580 2021-11-19T08:43:05.834 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.850+0000 7f4e8affd700 1 -- 172.21.15.23:0/1227237801 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2949 (secure 0 0 0) 0x7f4e8c04c440 con 0x7f4e9c0f5580 2021-11-19T08:43:05.835 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:05.835 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44},{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":55}],"filesystems":[{"mdsmap":{"epoch":57,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:04.533906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:05.837 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.851+0000 7f4ea0a64700 1 -- 172.21.15.23:0/1227237801 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4e8405c2a0 msgr2=0x7f4e8405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:05.837 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.851+0000 7f4ea0a64700 1 --2- 172.21.15.23:0/1227237801 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4e8405c2a0 0x7f4e8405e760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f4e90004e30 tx=0x7f4e90009b30).stop 2021-11-19T08:43:05.837 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.851+0000 7f4ea0a64700 1 -- 172.21.15.23:0/1227237801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e9c0f5580 msgr2=0x7f4e9c0604f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:05.837 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.851+0000 7f4ea0a64700 1 --2- 172.21.15.23:0/1227237801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e9c0f5580 0x7f4e9c0604f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f4e8c002700 tx=0x7f4e8c009e00).stop 2021-11-19T08:43:05.838 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.851+0000 7f4ea0a64700 1 -- 172.21.15.23:0/1227237801 shutdown_connections 2021-11-19T08:43:05.838 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.851+0000 7f4ea0a64700 1 --2- 172.21.15.23:0/1227237801 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4e8405c2a0 0x7f4e8405e760 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:05.838 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.851+0000 7f4ea0a64700 1 --2- 172.21.15.23:0/1227237801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e9c0f5580 0x7f4e9c0604f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:05.838 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.851+0000 7f4ea0a64700 1 --2- 172.21.15.23:0/1227237801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e9c0f6820 0x7f4e9c10bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:05.839 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.851+0000 7f4ea0a64700 1 -- 172.21.15.23:0/1227237801 >> 172.21.15.23:0/1227237801 conn(0x7f4e9c0f0ce0 msgr2=0x7f4e9c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:05.839 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.852+0000 7f4ea0a64700 1 -- 172.21.15.23:0/1227237801 shutdown_connections 2021-11-19T08:43:05.839 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:05.852+0000 7f4ea0a64700 1 -- 172.21.15.23:0/1227237801 wait complete. 2021-11-19T08:43:05.839 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 57 2021-11-19T08:43:05.848 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:05 smithi023 conmon[33490]: cluster 2021-11-19T08:43:04.540429+0000 mon.smithi023 (mon.0) 2021-11-19T08:43:05.848 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:05 smithi023 conmon[33490]: 696 : cluster [DBG] mds.? [v2:172.21.15.23:6826/2135889102,v1:172.21.15.23:6827/2135889102] up:resolve 2021-11-19T08:43:05.848 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:05 smithi023 conmon[33490]: cluster 2021-11-19T08:43:04.540470+0000 mon.smithi023 (mon.0) 697 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi023.wazanl=up:rejoin} 2 up:standby 2021-11-19T08:43:05.849 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:04.533906+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:05.850 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:43:06.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:06 smithi134 conmon[36363]: cluster 2021-11-19T08:43:04.869054+0000 mgr.smithi023.ehlpcl (mgr.14164) 272 : cluster 2021-11-19T08:43:06.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:06 smithi134 conmon[36363]: [DBG] pgmap v242: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-19T08:43:06.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:06 smithi134 conmon[36363]: audit 2021-11-19T08:43:05.850866+0000 mon.smithi023 (mon.0) 698 : audit [DBG] from='client.? 172.21.15.23:0/1227237801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:06.851 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:06.879 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:06 smithi023 conmon[33490]: cluster 2021-11-19T08:43:04.869054+0000 mgr.smithi023.ehlpcl 2021-11-19T08:43:06.880 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:06 smithi023 conmon[33490]: (mgr.14164) 272 : cluster [DBG] pgmap v242: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-19T08:43:06.880 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:06 smithi023 conmon[33490]: audit 2021-11-19T08:43:05.850866 2021-11-19T08:43:06.880 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:06 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 698 : audit [DBG] from='client.? 172.21.15.23:0/1227237801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:06.984 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.998+0000 7f498113b700 1 -- 172.21.15.23:0/2700203216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f497c0f7940 msgr2=0x7f497c0f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.998+0000 7f498113b700 1 --2- 172.21.15.23:0/2700203216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f497c0f7940 0x7f497c0f7da0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f4964004660 tx=0x7f4964009b30).stop 2021-11-19T08:43:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.998+0000 7f498113b700 1 -- 172.21.15.23:0/2700203216 shutdown_connections 2021-11-19T08:43:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.998+0000 7f498113b700 1 --2- 172.21.15.23:0/2700203216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f497c0f7940 0x7f497c0f7da0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.998+0000 7f498113b700 1 --2- 172.21.15.23:0/2700203216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f497c0f66a0 0x7f497c0f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.998+0000 7f498113b700 1 -- 172.21.15.23:0/2700203216 >> 172.21.15.23:0/2700203216 conn(0x7f497c0f1da0 msgr2=0x7f497c0f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.998+0000 7f498113b700 1 -- 172.21.15.23:0/2700203216 shutdown_connections 2021-11-19T08:43:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.998+0000 7f498113b700 1 -- 172.21.15.23:0/2700203216 wait complete. 2021-11-19T08:43:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.998+0000 7f498113b700 1 Processor -- start 2021-11-19T08:43:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f498113b700 1 -- start start 2021-11-19T08:43:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f498113b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f497c0f66a0 0x7f497c06b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f498113b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f497c0f7940 0x7f497c069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f498113b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c06bd70 con 0x7f497c0f7940 2021-11-19T08:43:06.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f498113b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c06a310 con 0x7f497c0f66a0 2021-11-19T08:43:06.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f497a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f497c0f7940 0x7f497c069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:06.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f497a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f497c0f7940 0x7f497c069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47716/0 (socket says 172.21.15.23:47716) 2021-11-19T08:43:06.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f497a59c700 1 -- 172.21.15.23:0/3138958984 learned_addr learned my addr 172.21.15.23:0/3138958984 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:06.989 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f497ad9d700 1 --2- 172.21.15.23:0/3138958984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f497c0f66a0 0x7f497c06b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:06.989 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f497a59c700 1 -- 172.21.15.23:0/3138958984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f497c0f66a0 msgr2=0x7f497c06b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:06.989 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f497a59c700 1 --2- 172.21.15.23:0/3138958984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f497c0f66a0 0x7f497c06b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:06.989 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f497a59c700 1 -- 172.21.15.23:0/3138958984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4964005040 con 0x7f497c0f7940 2021-11-19T08:43:06.989 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f497a59c700 1 --2- 172.21.15.23:0/3138958984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f497c0f7940 0x7f497c069e20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f4964009f50 tx=0x7f4964009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f49737fe700 1 -- 172.21.15.23:0/3138958984 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4964018070 con 0x7f497c0f7940 2021-11-19T08:43:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f498113b700 1 -- 172.21.15.23:0/3138958984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497c06a530 con 0x7f497c0f7940 2021-11-19T08:43:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f49737fe700 1 -- 172.21.15.23:0/3138958984 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4964013410 con 0x7f497c0f7940 2021-11-19T08:43:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:06.999+0000 7f49737fe700 1 -- 172.21.15.23:0/3138958984 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f496401c3c0 con 0x7f497c0f7940 2021-11-19T08:43:06.991 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.000+0000 7f498113b700 1 -- 172.21.15.23:0/3138958984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f497c06aa90 con 0x7f497c0f7940 2021-11-19T08:43:06.992 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.001+0000 7f498113b700 1 -- 172.21.15.23:0/3138958984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497c046330 con 0x7f497c0f7940 2021-11-19T08:43:06.992 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.001+0000 7f49737fe700 1 -- 172.21.15.23:0/3138958984 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f496401d420 con 0x7f497c0f7940 2021-11-19T08:43:06.993 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.001+0000 7f49737fe700 1 --2- 172.21.15.23:0/3138958984 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f496805c2a0 0x7f496805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:06.993 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.001+0000 7f49737fe700 1 -- 172.21.15.23:0/3138958984 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f4964077e40 con 0x7f497c0f7940 2021-11-19T08:43:06.993 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.001+0000 7f497ad9d700 1 --2- 172.21.15.23:0/3138958984 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f496805c2a0 0x7f496805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:06.993 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.002+0000 7f497ad9d700 1 --2- 172.21.15.23:0/3138958984 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f496805c2a0 0x7f496805e760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f497c0f77a0 tx=0x7f4974009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:06.993 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.005+0000 7f49737fe700 1 -- 172.21.15.23:0/3138958984 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f496404a800 con 0x7f497c0f7940 2021-11-19T08:43:07.183 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.199+0000 7f498113b700 1 -- 172.21.15.23:0/3138958984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f497c027a00 con 0x7f497c0f7940 2021-11-19T08:43:07.184 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.200+0000 7f49737fe700 1 -- 172.21.15.23:0/3138958984 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2949 (secure 0 0 0) 0x7f496404de20 con 0x7f497c0f7940 2021-11-19T08:43:07.185 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:07.185 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44},{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":55}],"filesystems":[{"mdsmap":{"epoch":57,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:04.533906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:07.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.201+0000 7f498113b700 1 -- 172.21.15.23:0/3138958984 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f496805c2a0 msgr2=0x7f496805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:07.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.201+0000 7f498113b700 1 --2- 172.21.15.23:0/3138958984 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f496805c2a0 0x7f496805e760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f497c0f77a0 tx=0x7f4974009040).stop 2021-11-19T08:43:07.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.201+0000 7f498113b700 1 -- 172.21.15.23:0/3138958984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f497c0f7940 msgr2=0x7f497c069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:07.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.201+0000 7f498113b700 1 --2- 172.21.15.23:0/3138958984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f497c0f7940 0x7f497c069e20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f4964009f50 tx=0x7f4964009b30).stop 2021-11-19T08:43:07.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.202+0000 7f498113b700 1 -- 172.21.15.23:0/3138958984 shutdown_connections 2021-11-19T08:43:07.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.202+0000 7f498113b700 1 --2- 172.21.15.23:0/3138958984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f497c0f66a0 0x7f497c06b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:07.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.202+0000 7f498113b700 1 --2- 172.21.15.23:0/3138958984 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f496805c2a0 0x7f496805e760 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:07.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.202+0000 7f498113b700 1 --2- 172.21.15.23:0/3138958984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f497c0f7940 0x7f497c069e20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:07.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.202+0000 7f498113b700 1 -- 172.21.15.23:0/3138958984 >> 172.21.15.23:0/3138958984 conn(0x7f497c0f1da0 msgr2=0x7f497c0fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:07.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.202+0000 7f498113b700 1 -- 172.21.15.23:0/3138958984 shutdown_connections 2021-11-19T08:43:07.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:07.202+0000 7f498113b700 1 -- 172.21.15.23:0/3138958984 wait complete. 2021-11-19T08:43:07.189 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 57 2021-11-19T08:43:07.198 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:04.533906+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:07.199 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:43:07.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:07 smithi023 conmon[33490]: audit 2021-11-19T08:43:07.200723+0000 mon.smithi023 (mon. 2021-11-19T08:43:07.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:07 smithi023 conmon[33490]: 0) 699 : audit [DBG] from='client.? 172.21.15.23:0/3138958984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:08.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:07 smithi134 conmon[36363]: audit 2021-11-19T08:43:07.200723+0000 mon.smithi023 (mon.0) 699 : audit [DBG] from='client.? 172.21.15.23:0/3138958984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:08.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:08.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.347+0000 7fc5a5f51700 1 -- 172.21.15.23:0/3241399026 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a00f7920 msgr2=0x7fc5a00f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:08.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.347+0000 7fc5a5f51700 1 --2- 172.21.15.23:0/3241399026 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a00f7920 0x7fc5a00f7d80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fc594004660 tx=0x7fc594009b30).stop 2021-11-19T08:43:08.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.347+0000 7fc5a5f51700 1 -- 172.21.15.23:0/3241399026 shutdown_connections 2021-11-19T08:43:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.347+0000 7fc5a5f51700 1 --2- 172.21.15.23:0/3241399026 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a00f7920 0x7fc5a00f7d80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.347+0000 7fc5a5f51700 1 --2- 172.21.15.23:0/3241399026 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5a00f6680 0x7fc5a00f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.347+0000 7fc5a5f51700 1 -- 172.21.15.23:0/3241399026 >> 172.21.15.23:0/3241399026 conn(0x7fc5a00f1de0 msgr2=0x7fc5a00f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:08.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.347+0000 7fc5a5f51700 1 -- 172.21.15.23:0/3241399026 shutdown_connections 2021-11-19T08:43:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.347+0000 7fc5a5f51700 1 -- 172.21.15.23:0/3241399026 wait complete. 2021-11-19T08:43:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.348+0000 7fc5a5f51700 1 Processor -- start 2021-11-19T08:43:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.348+0000 7fc5a5f51700 1 -- start start 2021-11-19T08:43:08.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.348+0000 7fc5a5f51700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a00f6680 0x7fc5a0106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.348+0000 7fc5a5f51700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5a00f7920 0x7fc5a0107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.348+0000 7fc5a5f51700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a0107930 con 0x7fc5a00f6680 2021-11-19T08:43:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.348+0000 7fc5a5f51700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a0107a70 con 0x7fc5a00f7920 2021-11-19T08:43:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.348+0000 7fc59f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a00f6680 0x7fc5a0106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:08.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.348+0000 7fc59f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a00f6680 0x7fc5a0106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47728/0 (socket says 172.21.15.23:47728) 2021-11-19T08:43:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.348+0000 7fc59f7fe700 1 -- 172.21.15.23:0/365884354 learned_addr learned my addr 172.21.15.23:0/365884354 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.348+0000 7fc59effd700 1 --2- 172.21.15.23:0/365884354 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5a00f7920 0x7fc5a0107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.349+0000 7fc59f7fe700 1 -- 172.21.15.23:0/365884354 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5a00f7920 msgr2=0x7fc5a0107380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:08.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.349+0000 7fc59f7fe700 1 --2- 172.21.15.23:0/365884354 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5a00f7920 0x7fc5a0107380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.349+0000 7fc59f7fe700 1 -- 172.21.15.23:0/365884354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc594005040 con 0x7fc5a00f6680 2021-11-19T08:43:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.349+0000 7fc59f7fe700 1 --2- 172.21.15.23:0/365884354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a00f6680 0x7fc5a0106e90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fc59000c790 tx=0x7fc590007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.349+0000 7fc587fff700 1 -- 172.21.15.23:0/365884354 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc59000e680 con 0x7fc5a00f6680 2021-11-19T08:43:08.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.349+0000 7fc587fff700 1 -- 172.21.15.23:0/365884354 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc59000ec90 con 0x7fc5a00f6680 2021-11-19T08:43:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.349+0000 7fc587fff700 1 -- 172.21.15.23:0/365884354 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc590012cb0 con 0x7fc5a00f6680 2021-11-19T08:43:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.349+0000 7fc5a5f51700 1 -- 172.21.15.23:0/365884354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a010c510 con 0x7fc5a00f6680 2021-11-19T08:43:08.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.349+0000 7fc5a5f51700 1 -- 172.21.15.23:0/365884354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a010cad0 con 0x7fc5a00f6680 2021-11-19T08:43:08.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.350+0000 7fc5a5f51700 1 -- 172.21.15.23:0/365884354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a0027a00 con 0x7fc5a00f6680 2021-11-19T08:43:08.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.350+0000 7fc587fff700 1 -- 172.21.15.23:0/365884354 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc590010070 con 0x7fc5a00f6680 2021-11-19T08:43:08.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.350+0000 7fc587fff700 1 --2- 172.21.15.23:0/365884354 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc58805c2a0 0x7fc58805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:08.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.351+0000 7fc587fff700 1 -- 172.21.15.23:0/365884354 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fc590076370 con 0x7fc5a00f6680 2021-11-19T08:43:08.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.351+0000 7fc59effd700 1 --2- 172.21.15.23:0/365884354 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc58805c2a0 0x7fc58805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:08.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.351+0000 7fc59effd700 1 --2- 172.21.15.23:0/365884354 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc58805c2a0 0x7fc58805e760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fc594013580 tx=0x7fc594009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:08.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.354+0000 7fc587fff700 1 -- 172.21.15.23:0/365884354 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc590048c70 con 0x7fc5a00f6680 2021-11-19T08:43:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.552+0000 7fc5a5f51700 1 -- 172.21.15.23:0/365884354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5a00fee10 con 0x7fc5a00f6680 2021-11-19T08:43:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.553+0000 7fc587fff700 1 -- 172.21.15.23:0/365884354 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2948 (secure 0 0 0) 0x7fc59004c290 con 0x7fc5a00f6680 2021-11-19T08:43:08.538 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:08.538 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44},{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:04.533906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.554+0000 7fc5a5f51700 1 -- 172.21.15.23:0/365884354 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc58805c2a0 msgr2=0x7fc58805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.554+0000 7fc5a5f51700 1 --2- 172.21.15.23:0/365884354 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc58805c2a0 0x7fc58805e760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fc594013580 tx=0x7fc594009b30).stop 2021-11-19T08:43:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.554+0000 7fc5a5f51700 1 -- 172.21.15.23:0/365884354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a00f6680 msgr2=0x7fc5a0106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.554+0000 7fc5a5f51700 1 --2- 172.21.15.23:0/365884354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a00f6680 0x7fc5a0106e90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fc59000c790 tx=0x7fc590007d90).stop 2021-11-19T08:43:08.541 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.554+0000 7fc5a5f51700 1 -- 172.21.15.23:0/365884354 shutdown_connections 2021-11-19T08:43:08.541 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.554+0000 7fc5a5f51700 1 --2- 172.21.15.23:0/365884354 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc58805c2a0 0x7fc58805e760 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:08.541 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.554+0000 7fc5a5f51700 1 --2- 172.21.15.23:0/365884354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a00f6680 0x7fc5a0106e90 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:08.541 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.554+0000 7fc5a5f51700 1 --2- 172.21.15.23:0/365884354 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5a00f7920 0x7fc5a0107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:08.542 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.554+0000 7fc5a5f51700 1 -- 172.21.15.23:0/365884354 >> 172.21.15.23:0/365884354 conn(0x7fc5a00f1de0 msgr2=0x7fc5a00fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:08.542 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.555+0000 7fc5a5f51700 1 -- 172.21.15.23:0/365884354 shutdown_connections 2021-11-19T08:43:08.542 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:08.555+0000 7fc5a5f51700 1 -- 172.21.15.23:0/365884354 wait complete. 2021-11-19T08:43:08.543 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 58 2021-11-19T08:43:08.551 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:04.533906+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:08.551 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:43:08.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:08 smithi023 conmon[33490]: cluster 2021-11-19T08:43:06.869538+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:43:08.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:08 smithi023 conmon[33490]: .14164) 273 : cluster [DBG] pgmap v243: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:08.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:08 smithi023 conmon[33490]: cluster 2021-11-19T08:43:07.566097+0000 mon.smithi023 (mon.0) 2021-11-19T08:43:08.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:08 smithi023 conmon[33490]: 700 : cluster [DBG] mds.? [v2:172.21.15.134:6824/464801254,v1:172.21.15.134:6825/464801254] up:standby 2021-11-19T08:43:08.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:08 smithi023 conmon[33490]: cluster 2021 2021-11-19T08:43:08.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:08 smithi023 conmon[33490]: -11-19T08:43:07.566170+0000 mon.smithi023 (mon 2021-11-19T08:43:08.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:08 smithi023 conmon[33490]: .0) 701 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi023.wazanl=up:rejoin} 2 up:standby 2021-11-19T08:43:08.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:08 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:43:08.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:08 smithi023 conmon[33490]: 11-19T08:43:08.553855+0000 2021-11-19T08:43:08.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:08 smithi023 conmon[33490]: mon.smithi023 (mon.0) 702 : audit [DBG] 2021-11-19T08:43:08.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:08 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/365884354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:09.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:08 smithi134 conmon[36363]: cluster 2021-11-19T08:43:06.869538+0000 mgr.smithi023.ehlpcl (mgr.14164) 273 : cluster [DBG] pgmap v243: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:09.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:08 smithi134 conmon[36363]: cluster 2021-11-19T08:43:07.566097+0000 mon.smithi023 (mon.0) 700 : cluster [DBG] mds.? [v2:172.21.15.134:6824/464801254,v1:172.21.15.134:6825/464801254] up:standby 2021-11-19T08:43:09.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:08 smithi134 conmon[36363]: cluster 2021-11-19T08 2021-11-19T08:43:09.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:08 smithi134 conmon[36363]: :43:07.566170+0000 mon.smithi023 (mon.0) 701 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi023.wazanl=up:rejoin} 2 up:standby 2021-11-19T08:43:09.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:08 smithi134 conmon[36363]: audit 2021-11 2021-11-19T08:43:09.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:08 smithi134 conmon[36363]: -19T08:43:08.553855+0000 mon.smithi023 ( 2021-11-19T08:43:09.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:08 smithi134 conmon[36363]: mon.0) 702 : audit [DBG] from='client.? 172.21.15.23:0/365884354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:09.558 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:09.694 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.708+0000 7f4bc4f40700 1 -- 172.21.15.23:0/3688656698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bc00f6840 msgr2=0x7f4bc00f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:09.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.708+0000 7f4bc4f40700 1 --2- 172.21.15.23:0/3688656698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bc00f6840 0x7f4bc00f6ca0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f4ba8004660 tx=0x7f4ba8009b30).stop 2021-11-19T08:43:09.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.708+0000 7f4bc4f40700 1 -- 172.21.15.23:0/3688656698 shutdown_connections 2021-11-19T08:43:09.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.708+0000 7f4bc4f40700 1 --2- 172.21.15.23:0/3688656698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bc00f6840 0x7f4bc00f6ca0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:09.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.708+0000 7f4bc4f40700 1 --2- 172.21.15.23:0/3688656698 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bc00f55a0 0x7f4bc00f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:09.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.708+0000 7f4bc4f40700 1 -- 172.21.15.23:0/3688656698 >> 172.21.15.23:0/3688656698 conn(0x7f4bc00f0ce0 msgr2=0x7f4bc00f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:09.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.708+0000 7f4bc4f40700 1 -- 172.21.15.23:0/3688656698 shutdown_connections 2021-11-19T08:43:09.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.708+0000 7f4bc4f40700 1 -- 172.21.15.23:0/3688656698 wait complete. 2021-11-19T08:43:09.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.708+0000 7f4bc4f40700 1 Processor -- start 2021-11-19T08:43:09.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.708+0000 7f4bc4f40700 1 -- start start 2021-11-19T08:43:09.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bc4f40700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bc00f55a0 0x7f4bc005ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:09.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bc4f40700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bc00f6840 0x7f4bc010bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:09.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bc4f40700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc010c110 con 0x7f4bc00f6840 2021-11-19T08:43:09.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bc4f40700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc010c250 con 0x7f4bc00f55a0 2021-11-19T08:43:09.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bbdd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bc00f6840 0x7f4bc010bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:09.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bbdd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bc00f6840 0x7f4bc010bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47742/0 (socket says 172.21.15.23:47742) 2021-11-19T08:43:09.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bbdd9b700 1 -- 172.21.15.23:0/960996437 learned_addr learned my addr 172.21.15.23:0/960996437 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:09.699 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bbe59c700 1 --2- 172.21.15.23:0/960996437 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bc00f55a0 0x7f4bc005ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:09.699 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bbdd9b700 1 -- 172.21.15.23:0/960996437 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bc00f55a0 msgr2=0x7f4bc005ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:09.699 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bbdd9b700 1 --2- 172.21.15.23:0/960996437 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bc00f55a0 0x7f4bc005ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:09.699 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bbdd9b700 1 -- 172.21.15.23:0/960996437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba8005040 con 0x7f4bc00f6840 2021-11-19T08:43:09.699 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bbdd9b700 1 --2- 172.21.15.23:0/960996437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bc00f6840 0x7f4bc010bc20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f4ba8009870 tx=0x7f4ba800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:09.700 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bb6ffd700 1 -- 172.21.15.23:0/960996437 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ba8019070 con 0x7f4bc00f6840 2021-11-19T08:43:09.700 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bb6ffd700 1 -- 172.21.15.23:0/960996437 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ba8007e70 con 0x7f4bc00f6840 2021-11-19T08:43:09.700 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bc4f40700 1 -- 172.21.15.23:0/960996437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc010c450 con 0x7f4bc00f6840 2021-11-19T08:43:09.701 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bb6ffd700 1 -- 172.21.15.23:0/960996437 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ba8014d00 con 0x7f4bc00f6840 2021-11-19T08:43:09.701 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.709+0000 7f4bc4f40700 1 -- 172.21.15.23:0/960996437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc010c9b0 con 0x7f4bc00f6840 2021-11-19T08:43:09.701 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.710+0000 7f4bc4f40700 1 -- 172.21.15.23:0/960996437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc0027a00 con 0x7f4bc00f6840 2021-11-19T08:43:09.702 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.711+0000 7f4bb6ffd700 1 -- 172.21.15.23:0/960996437 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4ba8025070 con 0x7f4bc00f6840 2021-11-19T08:43:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.711+0000 7f4bb6ffd700 1 --2- 172.21.15.23:0/960996437 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4bac05c2a0 0x7f4bac05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.711+0000 7f4bb6ffd700 1 -- 172.21.15.23:0/960996437 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f4ba80771c0 con 0x7f4bc00f6840 2021-11-19T08:43:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.711+0000 7f4bbe59c700 1 --2- 172.21.15.23:0/960996437 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4bac05c2a0 0x7f4bac05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.712+0000 7f4bbe59c700 1 --2- 172.21.15.23:0/960996437 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4bac05c2a0 0x7f4bac05e760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f4bb8004fd0 tx=0x7f4bb800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:09.704 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.715+0000 7f4bb6ffd700 1 -- 172.21.15.23:0/960996437 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ba8049b80 con 0x7f4bc00f6840 2021-11-19T08:43:09.896 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.912+0000 7f4bc4f40700 1 -- 172.21.15.23:0/960996437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bc00fdd30 con 0x7f4bc00f6840 2021-11-19T08:43:09.897 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.912+0000 7f4bb6ffd700 1 -- 172.21.15.23:0/960996437 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2948 (secure 0 0 0) 0x7f4ba804d1a0 con 0x7f4bc00f6840 2021-11-19T08:43:09.897 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:09.898 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44},{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:04.533906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:09.899 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.914+0000 7f4bc4f40700 1 -- 172.21.15.23:0/960996437 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4bac05c2a0 msgr2=0x7f4bac05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:09.899 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.914+0000 7f4bc4f40700 1 --2- 172.21.15.23:0/960996437 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4bac05c2a0 0x7f4bac05e760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f4bb8004fd0 tx=0x7f4bb800b040).stop 2021-11-19T08:43:09.899 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.914+0000 7f4bc4f40700 1 -- 172.21.15.23:0/960996437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bc00f6840 msgr2=0x7f4bc010bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:09.900 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.914+0000 7f4bc4f40700 1 --2- 172.21.15.23:0/960996437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bc00f6840 0x7f4bc010bc20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f4ba8009870 tx=0x7f4ba800b040).stop 2021-11-19T08:43:09.900 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.914+0000 7f4bc4f40700 1 -- 172.21.15.23:0/960996437 shutdown_connections 2021-11-19T08:43:09.900 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.914+0000 7f4bc4f40700 1 --2- 172.21.15.23:0/960996437 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bc00f55a0 0x7f4bc005ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:09.900 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.914+0000 7f4bc4f40700 1 --2- 172.21.15.23:0/960996437 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4bac05c2a0 0x7f4bac05e760 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:09.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.914+0000 7f4bc4f40700 1 --2- 172.21.15.23:0/960996437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bc00f6840 0x7f4bc010bc20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:09.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.914+0000 7f4bc4f40700 1 -- 172.21.15.23:0/960996437 >> 172.21.15.23:0/960996437 conn(0x7f4bc00f0ce0 msgr2=0x7f4bc00fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:09.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.914+0000 7f4bc4f40700 1 -- 172.21.15.23:0/960996437 shutdown_connections 2021-11-19T08:43:09.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:09.914+0000 7f4bc4f40700 1 -- 172.21.15.23:0/960996437 wait complete. 2021-11-19T08:43:09.902 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 58 2021-11-19T08:43:09.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:04.533906+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:09.911 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:43:10.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:10 smithi023 conmon[33490]: cluster 2021-11-19T08:43:08.870122+0000 mgr.smithi023.ehlpcl (mgr.14164) 274 : 2021-11-19T08:43:10.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:10 smithi023 conmon[33490]: cluster [DBG] pgmap v244: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:10.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:10 smithi023 conmon[33490]: audit 2021-11-19T08:43:09.913167+0000 mon.smithi023 (mon.0) 703 : audit [DBG] from='client.? 172.21.15.23:0/960996437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:10.912 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:11.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:10 smithi134 conmon[36363]: cluster 2021-11-19T08:43:08.870122+0000 mgr.smithi023.ehlpcl (mgr.14164) 274 : cluster [DBG] pgmap v244: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:11.293 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:10 smithi134 conmon[36363]: 2021-11-19T08:43:11.294 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:10 smithi134 conmon[36363]: audit 2021-11-19T08:43:09.913167+0000 mon.smithi023 (mon.0) 703 : audit [DBG] from='client.? 172.21.15.23:0/960996437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:11.298 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.059+0000 7f353259b700 1 -- 172.21.15.23:0/79193913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f352c0f7870 msgr2=0x7f352c0f7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:11.298 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.060+0000 7f353259b700 1 --2- 172.21.15.23:0/79193913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f352c0f7870 0x7f352c0f7cd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f351c004660 tx=0x7f351c009b30).stop 2021-11-19T08:43:11.298 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.060+0000 7f353259b700 1 -- 172.21.15.23:0/79193913 shutdown_connections 2021-11-19T08:43:11.298 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.060+0000 7f353259b700 1 --2- 172.21.15.23:0/79193913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f352c0f7870 0x7f352c0f7cd0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:11.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.060+0000 7f353259b700 1 --2- 172.21.15.23:0/79193913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f352c0f65d0 0x7f352c0f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:11.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.060+0000 7f353259b700 1 -- 172.21.15.23:0/79193913 >> 172.21.15.23:0/79193913 conn(0x7f352c0f1d10 msgr2=0x7f352c0f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:11.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.060+0000 7f353259b700 1 -- 172.21.15.23:0/79193913 shutdown_connections 2021-11-19T08:43:11.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.060+0000 7f353259b700 1 -- 172.21.15.23:0/79193913 wait complete. 2021-11-19T08:43:11.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.060+0000 7f353259b700 1 Processor -- start 2021-11-19T08:43:11.300 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.060+0000 7f353259b700 1 -- start start 2021-11-19T08:43:11.300 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.060+0000 7f353259b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f352c0f65d0 0x7f352c106d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:11.300 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.060+0000 7f353259b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f352c0f7870 0x7f352c107220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:11.300 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.060+0000 7f353259b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352c107820 con 0x7f352c0f7870 2021-11-19T08:43:11.301 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.060+0000 7f353259b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352c108030 con 0x7f352c0f65d0 2021-11-19T08:43:11.301 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.061+0000 7f3530d98700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f352c0f7870 0x7f352c107220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:11.301 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.061+0000 7f3530d98700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f352c0f7870 0x7f352c107220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47754/0 (socket says 172.21.15.23:47754) 2021-11-19T08:43:11.301 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.061+0000 7f3530d98700 1 -- 172.21.15.23:0/1660353285 learned_addr learned my addr 172.21.15.23:0/1660353285 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:11.301 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.061+0000 7f3531599700 1 --2- 172.21.15.23:0/1660353285 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f352c0f65d0 0x7f352c106d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:11.302 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.061+0000 7f3530d98700 1 -- 172.21.15.23:0/1660353285 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f352c0f65d0 msgr2=0x7f352c106d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:11.302 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.061+0000 7f3530d98700 1 --2- 172.21.15.23:0/1660353285 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f352c0f65d0 0x7f352c106d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:11.302 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.061+0000 7f3530d98700 1 -- 172.21.15.23:0/1660353285 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351c005040 con 0x7f352c0f7870 2021-11-19T08:43:11.302 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.061+0000 7f3530d98700 1 --2- 172.21.15.23:0/1660353285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f352c0f7870 0x7f352c107220 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f351c000c00 tx=0x7f351c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:11.303 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.061+0000 7f3521ffb700 1 -- 172.21.15.23:0/1660353285 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f351c019070 con 0x7f352c0f7870 2021-11-19T08:43:11.303 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.061+0000 7f353259b700 1 -- 172.21.15.23:0/1660353285 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f352c108250 con 0x7f352c0f7870 2021-11-19T08:43:11.303 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.061+0000 7f3521ffb700 1 -- 172.21.15.23:0/1660353285 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f351c007e70 con 0x7f352c0f7870 2021-11-19T08:43:11.303 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.061+0000 7f3521ffb700 1 -- 172.21.15.23:0/1660353285 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f351c014c90 con 0x7f352c0f7870 2021-11-19T08:43:11.303 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.061+0000 7f353259b700 1 -- 172.21.15.23:0/1660353285 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f352c1087b0 con 0x7f352c0f7870 2021-11-19T08:43:11.304 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.062+0000 7f353259b700 1 -- 172.21.15.23:0/1660353285 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f352c0f59c0 con 0x7f352c0f7870 2021-11-19T08:43:11.304 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.063+0000 7f3521ffb700 1 -- 172.21.15.23:0/1660353285 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f351c025070 con 0x7f352c0f7870 2021-11-19T08:43:11.304 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.063+0000 7f3521ffb700 1 --2- 172.21.15.23:0/1660353285 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f351805c300 0x7f351805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:11.304 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.063+0000 7f3521ffb700 1 -- 172.21.15.23:0/1660353285 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f351c077220 con 0x7f352c0f7870 2021-11-19T08:43:11.304 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.063+0000 7f3531599700 1 --2- 172.21.15.23:0/1660353285 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f351805c300 0x7f351805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:11.305 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.063+0000 7f3531599700 1 --2- 172.21.15.23:0/1660353285 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f351805c300 0x7f351805e7c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f352c106aa0 tx=0x7f352800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:11.305 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.067+0000 7f3521ffb700 1 -- 172.21.15.23:0/1660353285 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f351c049be0 con 0x7f352c0f7870 2021-11-19T08:43:11.305 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.263+0000 7f353259b700 1 -- 172.21.15.23:0/1660353285 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f352c0fee80 con 0x7f352c0f7870 2021-11-19T08:43:11.305 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.264+0000 7f3521ffb700 1 -- 172.21.15.23:0/1660353285 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2948 (secure 0 0 0) 0x7f352c0fee80 con 0x7f352c0f7870 2021-11-19T08:43:11.306 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:11.306 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44},{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:04.533906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:11.308 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.265+0000 7f353259b700 1 -- 172.21.15.23:0/1660353285 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f351805c300 msgr2=0x7f351805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:11.308 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.265+0000 7f353259b700 1 --2- 172.21.15.23:0/1660353285 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f351805c300 0x7f351805e7c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f352c106aa0 tx=0x7f352800b040).stop 2021-11-19T08:43:11.308 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.265+0000 7f353259b700 1 -- 172.21.15.23:0/1660353285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f352c0f7870 msgr2=0x7f352c107220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:11.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.265+0000 7f353259b700 1 --2- 172.21.15.23:0/1660353285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f352c0f7870 0x7f352c107220 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f351c000c00 tx=0x7f351c00b040).stop 2021-11-19T08:43:11.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.265+0000 7f353259b700 1 -- 172.21.15.23:0/1660353285 shutdown_connections 2021-11-19T08:43:11.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.265+0000 7f353259b700 1 --2- 172.21.15.23:0/1660353285 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f352c0f65d0 0x7f352c106d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:11.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.265+0000 7f353259b700 1 --2- 172.21.15.23:0/1660353285 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f351805c300 0x7f351805e7c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:11.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.265+0000 7f353259b700 1 --2- 172.21.15.23:0/1660353285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f352c0f7870 0x7f352c107220 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:11.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.265+0000 7f353259b700 1 -- 172.21.15.23:0/1660353285 >> 172.21.15.23:0/1660353285 conn(0x7f352c0f1d10 msgr2=0x7f352c0fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:11.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.265+0000 7f353259b700 1 -- 172.21.15.23:0/1660353285 shutdown_connections 2021-11-19T08:43:11.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:11.265+0000 7f353259b700 1 -- 172.21.15.23:0/1660353285 wait complete. 2021-11-19T08:43:11.310 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 58 2021-11-19T08:43:11.311 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:04.533906+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:11.311 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:43:11.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:11 smithi023 conmon[33490]: audit 2021-11-19T08:43:11.264857+0000 mon.smithi023 (mon.0) 704 2021-11-19T08:43:11.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:11 smithi023 conmon[33490]: : audit [DBG] from='client.? 172.21.15.23:0/1660353285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:12.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:11 smithi134 conmon[36363]: audit 2021-11-19T08:43:11.264857+0000 mon.smithi023 (mon.0) 704 : audit [DBG] from='client.? 172.21.15.23:0/1660353285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:12.312 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:12.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.460+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/222815365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f6810 msgr2=0x7f4c380f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:12.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.460+0000 7f4c3f0d8700 1 --2- 172.21.15.23:0/222815365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f6810 0x7f4c380f6c70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f4c2c004660 tx=0x7f4c2c009b30).stop 2021-11-19T08:43:12.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.460+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/222815365 shutdown_connections 2021-11-19T08:43:12.453 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.460+0000 7f4c3f0d8700 1 --2- 172.21.15.23:0/222815365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f6810 0x7f4c380f6c70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:12.453 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.460+0000 7f4c3f0d8700 1 --2- 172.21.15.23:0/222815365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c380f5570 0x7f4c380f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:12.454 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.460+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/222815365 >> 172.21.15.23:0/222815365 conn(0x7f4c380f0cf0 msgr2=0x7f4c380f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:12.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.460+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/222815365 shutdown_connections 2021-11-19T08:43:12.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.460+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/222815365 wait complete. 2021-11-19T08:43:12.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.461+0000 7f4c3f0d8700 1 Processor -- start 2021-11-19T08:43:12.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.461+0000 7f4c3f0d8700 1 -- start start 2021-11-19T08:43:12.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.461+0000 7f4c3f0d8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f5570 0x7f4c380604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:12.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.461+0000 7f4c3f0d8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c380f6810 0x7f4c3810bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:12.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.461+0000 7f4c3f0d8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3810c170 con 0x7f4c380f5570 2021-11-19T08:43:12.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.461+0000 7f4c3f0d8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3810c2b0 con 0x7f4c380f6810 2021-11-19T08:43:12.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.461+0000 7f4c3ce74700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f5570 0x7f4c380604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:12.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.461+0000 7f4c3ce74700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f5570 0x7f4c380604f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47762/0 (socket says 172.21.15.23:47762) 2021-11-19T08:43:12.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.461+0000 7f4c3ce74700 1 -- 172.21.15.23:0/273153472 learned_addr learned my addr 172.21.15.23:0/273153472 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:12.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.462+0000 7f4c37fff700 1 --2- 172.21.15.23:0/273153472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c380f6810 0x7f4c3810bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:12.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.462+0000 7f4c3ce74700 1 -- 172.21.15.23:0/273153472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c380f6810 msgr2=0x7f4c3810bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:12.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.462+0000 7f4c3ce74700 1 --2- 172.21.15.23:0/273153472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c380f6810 0x7f4c3810bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:12.460 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.462+0000 7f4c3ce74700 1 -- 172.21.15.23:0/273153472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c005040 con 0x7f4c380f5570 2021-11-19T08:43:12.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.462+0000 7f4c37fff700 1 --2- 172.21.15.23:0/273153472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c380f6810 0x7f4c3810bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:43:12.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.462+0000 7f4c3ce74700 1 --2- 172.21.15.23:0/273153472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f5570 0x7f4c380604f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f4c3000c810 tx=0x7f4c30004d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:12.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.462+0000 7f4c357fa700 1 -- 172.21.15.23:0/273153472 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c3000a700 con 0x7f4c380f5570 2021-11-19T08:43:12.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.462+0000 7f4c357fa700 1 -- 172.21.15.23:0/273153472 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c3000ad10 con 0x7f4c380f5570 2021-11-19T08:43:12.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.462+0000 7f4c357fa700 1 -- 172.21.15.23:0/273153472 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c30012e00 con 0x7f4c380f5570 2021-11-19T08:43:12.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.462+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/273153472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3810c510 con 0x7f4c380f5570 2021-11-19T08:43:12.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.462+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/273153472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3810caa0 con 0x7f4c380f5570 2021-11-19T08:43:12.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.463+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/273153472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c38046330 con 0x7f4c380f5570 2021-11-19T08:43:12.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.463+0000 7f4c357fa700 1 -- 172.21.15.23:0/273153472 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4c30019030 con 0x7f4c380f5570 2021-11-19T08:43:12.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.464+0000 7f4c357fa700 1 --2- 172.21.15.23:0/273153472 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c2005c2a0 0x7f4c2005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:12.464 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.464+0000 7f4c357fa700 1 -- 172.21.15.23:0/273153472 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f4c300761a0 con 0x7f4c380f5570 2021-11-19T08:43:12.465 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.464+0000 7f4c37fff700 1 --2- 172.21.15.23:0/273153472 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c2005c2a0 0x7f4c2005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:12.465 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.464+0000 7f4c37fff700 1 --2- 172.21.15.23:0/273153472 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c2005c2a0 0x7f4c2005e760 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4c2c009d60 tx=0x7f4c2c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:12.466 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.468+0000 7f4c357fa700 1 -- 172.21.15.23:0/273153472 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c300462b0 con 0x7f4c380f5570 2021-11-19T08:43:12.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:12 smithi023 conmon[33490]: cluster 2021-11-19T08:43:10. 2021-11-19T08:43:12.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:12 smithi023 conmon[33490]: 870659+0000 mgr.smithi023.ehlpcl (mgr.14164) 275 : cluster [DBG] pgmap v245: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:12.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.663+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/273153472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c38027a00 con 0x7f4c380f5570 2021-11-19T08:43:12.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.664+0000 7f4c357fa700 1 -- 172.21.15.23:0/273153472 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2948 (secure 0 0 0) 0x7f4c30017070 con 0x7f4c380f5570 2021-11-19T08:43:12.649 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:12.649 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44},{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:04.533906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:12.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.665+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/273153472 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c2005c2a0 msgr2=0x7f4c2005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:12.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.665+0000 7f4c3f0d8700 1 --2- 172.21.15.23:0/273153472 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c2005c2a0 0x7f4c2005e760 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4c2c009d60 tx=0x7f4c2c009b30).stop 2021-11-19T08:43:12.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.665+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/273153472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f5570 msgr2=0x7f4c380604f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:12.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.665+0000 7f4c3f0d8700 1 --2- 172.21.15.23:0/273153472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f5570 0x7f4c380604f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f4c3000c810 tx=0x7f4c30004d90).stop 2021-11-19T08:43:12.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.666+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/273153472 shutdown_connections 2021-11-19T08:43:12.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.666+0000 7f4c3f0d8700 1 --2- 172.21.15.23:0/273153472 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c2005c2a0 0x7f4c2005e760 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:12.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.666+0000 7f4c3f0d8700 1 --2- 172.21.15.23:0/273153472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f5570 0x7f4c380604f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:12.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.666+0000 7f4c3f0d8700 1 --2- 172.21.15.23:0/273153472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c380f6810 0x7f4c3810bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:12.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.666+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/273153472 >> 172.21.15.23:0/273153472 conn(0x7f4c380f0cf0 msgr2=0x7f4c380fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:12.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.666+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/273153472 shutdown_connections 2021-11-19T08:43:12.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:12.666+0000 7f4c3f0d8700 1 -- 172.21.15.23:0/273153472 wait complete. 2021-11-19T08:43:12.653 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 58 2021-11-19T08:43:12.662 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:04.533906+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:12.663 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:43:13.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:12 smithi134 conmon[36363]: cluster 2021-11-19T08:43:10.870659+0000 mgr.smithi023.ehlpcl (mgr.14164) 275 : cluster 2021-11-19T08:43:13.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:12 smithi134 conmon[36363]: [DBG] pgmap v245: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:13.664 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:13.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.818+0000 7fc061a77700 1 -- 172.21.15.23:0/2103248502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc05c0f7920 msgr2=0x7fc05c0f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:13.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.818+0000 7fc061a77700 1 --2- 172.21.15.23:0/2103248502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc05c0f7920 0x7fc05c0f7d80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fc050004660 tx=0x7fc050009b30).stop 2021-11-19T08:43:13.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.818+0000 7fc061a77700 1 -- 172.21.15.23:0/2103248502 shutdown_connections 2021-11-19T08:43:13.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.818+0000 7fc061a77700 1 --2- 172.21.15.23:0/2103248502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc05c0f7920 0x7fc05c0f7d80 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:13.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.818+0000 7fc061a77700 1 --2- 172.21.15.23:0/2103248502 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc05c0f6680 0x7fc05c0f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:13.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.818+0000 7fc061a77700 1 -- 172.21.15.23:0/2103248502 >> 172.21.15.23:0/2103248502 conn(0x7fc05c0f1de0 msgr2=0x7fc05c0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:13.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.818+0000 7fc061a77700 1 -- 172.21.15.23:0/2103248502 shutdown_connections 2021-11-19T08:43:13.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.818+0000 7fc061a77700 1 -- 172.21.15.23:0/2103248502 wait complete. 2021-11-19T08:43:13.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.819+0000 7fc061a77700 1 Processor -- start 2021-11-19T08:43:13.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.819+0000 7fc061a77700 1 -- start start 2021-11-19T08:43:13.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.819+0000 7fc061a77700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc05c0f6680 0x7fc05c106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:13.808 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.819+0000 7fc061a77700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc05c0f7920 0x7fc05c107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:13.808 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.819+0000 7fc061a77700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05c107930 con 0x7fc05c0f6680 2021-11-19T08:43:13.808 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.819+0000 7fc061a77700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05c107a70 con 0x7fc05c0f7920 2021-11-19T08:43:13.809 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.819+0000 7fc05b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc05c0f6680 0x7fc05c106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:13.813 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.819+0000 7fc05b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc05c0f6680 0x7fc05c106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47772/0 (socket says 172.21.15.23:47772) 2021-11-19T08:43:13.814 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.819+0000 7fc05b7fe700 1 -- 172.21.15.23:0/1451379315 learned_addr learned my addr 172.21.15.23:0/1451379315 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:13.814 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.819+0000 7fc05affd700 1 --2- 172.21.15.23:0/1451379315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc05c0f7920 0x7fc05c107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:13.814 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.820+0000 7fc05b7fe700 1 -- 172.21.15.23:0/1451379315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc05c0f7920 msgr2=0x7fc05c107380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:13.815 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.820+0000 7fc05b7fe700 1 --2- 172.21.15.23:0/1451379315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc05c0f7920 0x7fc05c107380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:13.815 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.820+0000 7fc05b7fe700 1 -- 172.21.15.23:0/1451379315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc050005040 con 0x7fc05c0f6680 2021-11-19T08:43:13.816 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.820+0000 7fc05b7fe700 1 --2- 172.21.15.23:0/1451379315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc05c0f6680 0x7fc05c106e90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fc04c002700 tx=0x7fc04c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:13.816 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.820+0000 7fc043fff700 1 -- 172.21.15.23:0/1451379315 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc04c0079c0 con 0x7fc05c0f6680 2021-11-19T08:43:13.817 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.820+0000 7fc043fff700 1 -- 172.21.15.23:0/1451379315 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc04c00f040 con 0x7fc05c0f6680 2021-11-19T08:43:13.817 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.820+0000 7fc061a77700 1 -- 172.21.15.23:0/1451379315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc05c10c510 con 0x7fc05c0f6680 2021-11-19T08:43:13.817 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.820+0000 7fc043fff700 1 -- 172.21.15.23:0/1451379315 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc04c014860 con 0x7fc05c0f6680 2021-11-19T08:43:13.817 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.820+0000 7fc061a77700 1 -- 172.21.15.23:0/1451379315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc05c10cad0 con 0x7fc05c0f6680 2021-11-19T08:43:13.817 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.821+0000 7fc061a77700 1 -- 172.21.15.23:0/1451379315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc05c027a00 con 0x7fc05c0f6680 2021-11-19T08:43:13.818 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.821+0000 7fc043fff700 1 -- 172.21.15.23:0/1451379315 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc04c01b060 con 0x7fc05c0f6680 2021-11-19T08:43:13.818 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.822+0000 7fc043fff700 1 --2- 172.21.15.23:0/1451379315 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc04405c2a0 0x7fc04405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:13.818 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.822+0000 7fc043fff700 1 -- 172.21.15.23:0/1451379315 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fc04c076140 con 0x7fc05c0f6680 2021-11-19T08:43:13.818 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.822+0000 7fc05affd700 1 --2- 172.21.15.23:0/1451379315 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc04405c2a0 0x7fc04405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:13.819 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.822+0000 7fc05affd700 1 --2- 172.21.15.23:0/1451379315 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc04405c2a0 0x7fc04405e760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc050009f50 tx=0x7fc050009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:13.819 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:13.826+0000 7fc043fff700 1 -- 172.21.15.23:0/1451379315 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc04c048a40 con 0x7fc05c0f6680 2021-11-19T08:43:13.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:13 smithi023 conmon[33490]: audit 2021-11-19T08:43:12.665074+0000 mon.smithi023 (mon.0) 705 : audit [DBG] from='client.? 172.21.15.23:0/273153472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:14.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:14.024+0000 7fc061a77700 1 -- 172.21.15.23:0/1451379315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc05c0fee10 con 0x7fc05c0f6680 2021-11-19T08:43:14.010 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:14.025+0000 7fc043fff700 1 -- 172.21.15.23:0/1451379315 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2948 (secure 0 0 0) 0x7fc04c04c060 con 0x7fc05c0f6680 2021-11-19T08:43:14.011 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:14.011 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44},{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:04.533906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:14.014 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:14.026+0000 7fc061a77700 1 -- 172.21.15.23:0/1451379315 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc04405c2a0 msgr2=0x7fc04405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:14.014 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:14.026+0000 7fc061a77700 1 --2- 172.21.15.23:0/1451379315 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc04405c2a0 0x7fc04405e760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc050009f50 tx=0x7fc050009b30).stop 2021-11-19T08:43:14.014 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:14.026+0000 7fc061a77700 1 -- 172.21.15.23:0/1451379315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc05c0f6680 msgr2=0x7fc05c106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:14.015 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:14.026+0000 7fc061a77700 1 --2- 172.21.15.23:0/1451379315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc05c0f6680 0x7fc05c106e90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fc04c002700 tx=0x7fc04c009d80).stop 2021-11-19T08:43:14.015 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:14.027+0000 7fc061a77700 1 -- 172.21.15.23:0/1451379315 shutdown_connections 2021-11-19T08:43:14.015 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:14.027+0000 7fc061a77700 1 --2- 172.21.15.23:0/1451379315 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc04405c2a0 0x7fc04405e760 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:14.015 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:14.027+0000 7fc061a77700 1 --2- 172.21.15.23:0/1451379315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc05c0f6680 0x7fc05c106e90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:14.017 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:14.027+0000 7fc061a77700 1 --2- 172.21.15.23:0/1451379315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc05c0f7920 0x7fc05c107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:14.017 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:14.027+0000 7fc061a77700 1 -- 172.21.15.23:0/1451379315 >> 172.21.15.23:0/1451379315 conn(0x7fc05c0f1de0 msgr2=0x7fc05c0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:14.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:14.027+0000 7fc061a77700 1 -- 172.21.15.23:0/1451379315 shutdown_connections 2021-11-19T08:43:14.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:14.027+0000 7fc061a77700 1 -- 172.21.15.23:0/1451379315 wait complete. 2021-11-19T08:43:14.018 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 58 2021-11-19T08:43:14.024 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:04.533906+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:14.024 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:43:14.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:13 smithi134 conmon[36363]: audit 2021-11-19T08:43:12.665074+0000 mon.smithi023 (mon.0) 705 : audit [DBG] from='client.? 172.21.15.23:0/273153472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:14.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:14 smithi023 conmon[33490]: cluster 2021-11-19T08:43:12.871168+0000 mgr.smithi023.ehlpcl (mgr.14164) 276 : cluster [DBG] pgmap v246: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:14.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:14 smithi023 conmon[33490]: audit 2021-11-19T08:43:14.026033+0000 mon.smithi023 (mon.0) 706 : audit [DBG] from='client.? 172.21.15.23:0/1451379315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:15.027 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:15.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:14 smithi134 conmon[36363]: cluster 2021-11-19T08:43:12.871168+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:43:15.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:14 smithi134 conmon[36363]: 276 : cluster [DBG] pgmap v246: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:15.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:14 smithi134 conmon[36363]: audit 2021-11-19T08:43:14.026033+0000 mon.smithi023 (mon.0) 706 : audit [DBG] from='client.? 172.21.15.23:0/1451379315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:15.152 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.165+0000 7f073d63c700 1 -- 172.21.15.23:0/3048784360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07380f65b0 msgr2=0x7f07380f69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:15.154 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.165+0000 7f073d63c700 1 --2- 172.21.15.23:0/3048784360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07380f65b0 0x7f07380f69d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f0728004660 tx=0x7f0728009b30).stop 2021-11-19T08:43:15.154 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.165+0000 7f073d63c700 1 -- 172.21.15.23:0/3048784360 shutdown_connections 2021-11-19T08:43:15.155 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.165+0000 7f073d63c700 1 --2- 172.21.15.23:0/3048784360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07380f7850 0x7f07380f7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:15.155 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.165+0000 7f073d63c700 1 --2- 172.21.15.23:0/3048784360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07380f65b0 0x7f07380f69d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:15.155 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.165+0000 7f073d63c700 1 -- 172.21.15.23:0/3048784360 >> 172.21.15.23:0/3048784360 conn(0x7f07380f1d10 msgr2=0x7f07380f4130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:15.155 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.165+0000 7f073d63c700 1 -- 172.21.15.23:0/3048784360 shutdown_connections 2021-11-19T08:43:15.156 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.165+0000 7f073d63c700 1 -- 172.21.15.23:0/3048784360 wait complete. 2021-11-19T08:43:15.156 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.166+0000 7f073d63c700 1 Processor -- start 2021-11-19T08:43:15.156 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.166+0000 7f073d63c700 1 -- start start 2021-11-19T08:43:15.157 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.166+0000 7f073d63c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07380f65b0 0x7f0738108fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:15.157 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.166+0000 7f073d63c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07380f7850 0x7f07381094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:15.158 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.166+0000 7f073d63c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0738109a60 con 0x7f07380f7850 2021-11-19T08:43:15.158 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.166+0000 7f073d63c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0738109ba0 con 0x7f07380f65b0 2021-11-19T08:43:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.166+0000 7f07367fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07380f7850 0x7f07381094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.166+0000 7f07367fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07380f7850 0x7f07381094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47784/0 (socket says 172.21.15.23:47784) 2021-11-19T08:43:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.166+0000 7f07367fc700 1 -- 172.21.15.23:0/659059175 learned_addr learned my addr 172.21.15.23:0/659059175 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:15.159 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.167+0000 7f0736ffd700 1 --2- 172.21.15.23:0/659059175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07380f65b0 0x7f0738108fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.167+0000 7f07367fc700 1 -- 172.21.15.23:0/659059175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07380f65b0 msgr2=0x7f0738108fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.167+0000 7f07367fc700 1 --2- 172.21.15.23:0/659059175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07380f65b0 0x7f0738108fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.167+0000 7f07367fc700 1 -- 172.21.15.23:0/659059175 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0728005040 con 0x7f07380f7850 2021-11-19T08:43:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.167+0000 7f07367fc700 1 --2- 172.21.15.23:0/659059175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07380f7850 0x7f07381094b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f072c0026d0 tx=0x7f072c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:15.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.167+0000 7f071f7fe700 1 -- 172.21.15.23:0/659059175 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f072c009320 con 0x7f07380f7850 2021-11-19T08:43:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.167+0000 7f071f7fe700 1 -- 172.21.15.23:0/659059175 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f072c00f040 con 0x7f07380f7850 2021-11-19T08:43:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.167+0000 7f073d63c700 1 -- 172.21.15.23:0/659059175 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073810a540 con 0x7f07380f7850 2021-11-19T08:43:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.167+0000 7f071f7fe700 1 -- 172.21.15.23:0/659059175 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f072c0147e0 con 0x7f07380f7850 2021-11-19T08:43:15.161 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.167+0000 7f073d63c700 1 -- 172.21.15.23:0/659059175 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073810ab00 con 0x7f07380f7850 2021-11-19T08:43:15.162 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.168+0000 7f073d63c700 1 -- 172.21.15.23:0/659059175 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0738045ad0 con 0x7f07380f7850 2021-11-19T08:43:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.168+0000 7f071f7fe700 1 -- 172.21.15.23:0/659059175 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f072c009480 con 0x7f07380f7850 2021-11-19T08:43:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.169+0000 7f071f7fe700 1 --2- 172.21.15.23:0/659059175 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f072005c300 0x7f072005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.169+0000 7f071f7fe700 1 -- 172.21.15.23:0/659059175 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f072c0767b0 con 0x7f07380f7850 2021-11-19T08:43:15.164 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.169+0000 7f0736ffd700 1 --2- 172.21.15.23:0/659059175 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f072005c300 0x7f072005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:15.165 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.169+0000 7f0736ffd700 1 --2- 172.21.15.23:0/659059175 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f072005c300 0x7f072005e7c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f0728004e30 tx=0x7f0728009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:15.165 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.173+0000 7f071f7fe700 1 -- 172.21.15.23:0/659059175 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f072c0491f0 con 0x7f07380f7850 2021-11-19T08:43:15.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.378+0000 7f073d63c700 1 -- 172.21.15.23:0/659059175 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0738027a00 con 0x7f07380f7850 2021-11-19T08:43:15.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.379+0000 7f071f7fe700 1 -- 172.21.15.23:0/659059175 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2948 (secure 0 0 0) 0x7f072c04c810 con 0x7f07380f7850 2021-11-19T08:43:15.365 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:15.365 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44},{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:04.533906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:15.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.380+0000 7f073d63c700 1 -- 172.21.15.23:0/659059175 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f072005c300 msgr2=0x7f072005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:15.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.380+0000 7f073d63c700 1 --2- 172.21.15.23:0/659059175 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f072005c300 0x7f072005e7c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f0728004e30 tx=0x7f0728009b30).stop 2021-11-19T08:43:15.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.380+0000 7f073d63c700 1 -- 172.21.15.23:0/659059175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07380f7850 msgr2=0x7f07381094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:15.369 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.380+0000 7f073d63c700 1 --2- 172.21.15.23:0/659059175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07380f7850 0x7f07381094b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f072c0026d0 tx=0x7f072c009de0).stop 2021-11-19T08:43:15.369 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.380+0000 7f073d63c700 1 -- 172.21.15.23:0/659059175 shutdown_connections 2021-11-19T08:43:15.370 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.380+0000 7f073d63c700 1 --2- 172.21.15.23:0/659059175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07380f65b0 0x7f0738108fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:15.370 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.380+0000 7f073d63c700 1 --2- 172.21.15.23:0/659059175 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f072005c300 0x7f072005e7c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:15.371 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.380+0000 7f073d63c700 1 --2- 172.21.15.23:0/659059175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07380f7850 0x7f07381094b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:15.371 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.380+0000 7f073d63c700 1 -- 172.21.15.23:0/659059175 >> 172.21.15.23:0/659059175 conn(0x7f07380f1d10 msgr2=0x7f07380f9290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:15.371 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.381+0000 7f073d63c700 1 -- 172.21.15.23:0/659059175 shutdown_connections 2021-11-19T08:43:15.372 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:15.381+0000 7f073d63c700 1 -- 172.21.15.23:0/659059175 wait complete. 2021-11-19T08:43:15.373 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 58 2021-11-19T08:43:15.377 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:04.533906+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:15.378 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:43:15.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:15 smithi023 conmon[33490]: audit 2021-11-19T08:43:15.379761+0000 mon.smithi023 (mon.0) 707 : audit [DBG] from='client.? 172.21.15.23:0/659059175' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:16.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:15 smithi134 conmon[36363]: audit 2021-11-19T08:43:15.379761+0000 mon.smithi023 (mon.0) 707 : 2021-11-19T08:43:16.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:15 smithi134 conmon[36363]: audit [DBG] from='client.? 172.21.15.23:0/659059175' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:16.380 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:16.517 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.530+0000 7fea22fe1700 1 -- 172.21.15.23:0/2361191900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea1c0f5580 msgr2=0x7fea1c0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:16.517 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.530+0000 7fea22fe1700 1 --2- 172.21.15.23:0/2361191900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea1c0f5580 0x7fea1c0f59a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fea10004660 tx=0x7fea10009b30).stop 2021-11-19T08:43:16.517 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.530+0000 7fea22fe1700 1 -- 172.21.15.23:0/2361191900 shutdown_connections 2021-11-19T08:43:16.518 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.530+0000 7fea22fe1700 1 --2- 172.21.15.23:0/2361191900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea1c0f6820 0x7fea1c0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:16.518 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.530+0000 7fea22fe1700 1 --2- 172.21.15.23:0/2361191900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea1c0f5580 0x7fea1c0f59a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:16.518 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.530+0000 7fea22fe1700 1 -- 172.21.15.23:0/2361191900 >> 172.21.15.23:0/2361191900 conn(0x7fea1c0f0ce0 msgr2=0x7fea1c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.530+0000 7fea22fe1700 1 -- 172.21.15.23:0/2361191900 shutdown_connections 2021-11-19T08:43:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.530+0000 7fea22fe1700 1 -- 172.21.15.23:0/2361191900 wait complete. 2021-11-19T08:43:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.531+0000 7fea22fe1700 1 Processor -- start 2021-11-19T08:43:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.531+0000 7fea22fe1700 1 -- start start 2021-11-19T08:43:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.531+0000 7fea22fe1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea1c0f5580 0x7fea1c105d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:16.520 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.531+0000 7fea22fe1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea1c0f6820 0x7fea1c106210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:16.520 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.531+0000 7fea22fe1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c1067c0 con 0x7fea1c0f5580 2021-11-19T08:43:16.520 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.531+0000 7fea22fe1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c106900 con 0x7fea1c0f6820 2021-11-19T08:43:16.520 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.531+0000 7fea20d7d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea1c0f5580 0x7fea1c105d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:16.521 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.531+0000 7fea20d7d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea1c0f5580 0x7fea1c105d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47800/0 (socket says 172.21.15.23:47800) 2021-11-19T08:43:16.521 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.531+0000 7fea20d7d700 1 -- 172.21.15.23:0/1227249038 learned_addr learned my addr 172.21.15.23:0/1227249038 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:16.521 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.531+0000 7fea1bfff700 1 --2- 172.21.15.23:0/1227249038 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea1c0f6820 0x7fea1c106210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:16.521 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.532+0000 7fea20d7d700 1 -- 172.21.15.23:0/1227249038 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea1c0f6820 msgr2=0x7fea1c106210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:16.521 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.532+0000 7fea20d7d700 1 --2- 172.21.15.23:0/1227249038 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea1c0f6820 0x7fea1c106210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:16.522 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.532+0000 7fea20d7d700 1 -- 172.21.15.23:0/1227249038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea10005040 con 0x7fea1c0f5580 2021-11-19T08:43:16.522 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.532+0000 7fea20d7d700 1 --2- 172.21.15.23:0/1227249038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea1c0f5580 0x7fea1c105d20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fea10000c00 tx=0x7fea1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:16.522 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.532+0000 7fea197fa700 1 -- 172.21.15.23:0/1227249038 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea1001b070 con 0x7fea1c0f5580 2021-11-19T08:43:16.522 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.532+0000 7fea197fa700 1 -- 172.21.15.23:0/1227249038 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea10007e70 con 0x7fea1c0f5580 2021-11-19T08:43:16.523 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.532+0000 7fea22fe1700 1 -- 172.21.15.23:0/1227249038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1c05fa20 con 0x7fea1c0f5580 2021-11-19T08:43:16.523 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.532+0000 7fea197fa700 1 -- 172.21.15.23:0/1227249038 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea10014d40 con 0x7fea1c0f5580 2021-11-19T08:43:16.523 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.532+0000 7fea22fe1700 1 -- 172.21.15.23:0/1227249038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1c05ffe0 con 0x7fea1c0f5580 2021-11-19T08:43:16.524 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.533+0000 7fea22fe1700 1 -- 172.21.15.23:0/1227249038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1c046330 con 0x7fea1c0f5580 2021-11-19T08:43:16.524 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.533+0000 7fea197fa700 1 -- 172.21.15.23:0/1227249038 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fea10027070 con 0x7fea1c0f5580 2021-11-19T08:43:16.525 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.534+0000 7fea197fa700 1 --2- 172.21.15.23:0/1227249038 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fea0c05c2a0 0x7fea0c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:16.525 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.534+0000 7fea197fa700 1 -- 172.21.15.23:0/1227249038 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fea10078e90 con 0x7fea1c0f5580 2021-11-19T08:43:16.525 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.534+0000 7fea1bfff700 1 --2- 172.21.15.23:0/1227249038 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fea0c05c2a0 0x7fea0c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:16.525 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.534+0000 7fea1bfff700 1 --2- 172.21.15.23:0/1227249038 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fea0c05c2a0 0x7fea0c05e760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fea08009fd0 tx=0x7fea0800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:16.526 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.538+0000 7fea197fa700 1 -- 172.21.15.23:0/1227249038 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea1004b850 con 0x7fea1c0f5580 2021-11-19T08:43:16.723 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.739+0000 7fea22fe1700 1 -- 172.21.15.23:0/1227249038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea1c10b340 con 0x7fea1c0f5580 2021-11-19T08:43:16.724 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.739+0000 7fea197fa700 1 -- 172.21.15.23:0/1227249038 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2948 (secure 0 0 0) 0x7fea1004ee70 con 0x7fea1c0f5580 2021-11-19T08:43:16.724 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:16.725 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44},{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:04.533906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:16.726 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.741+0000 7fea22fe1700 1 -- 172.21.15.23:0/1227249038 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fea0c05c2a0 msgr2=0x7fea0c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:16.726 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.741+0000 7fea22fe1700 1 --2- 172.21.15.23:0/1227249038 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fea0c05c2a0 0x7fea0c05e760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fea08009fd0 tx=0x7fea0800b040).stop 2021-11-19T08:43:16.727 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.741+0000 7fea22fe1700 1 -- 172.21.15.23:0/1227249038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea1c0f5580 msgr2=0x7fea1c105d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:16.727 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.741+0000 7fea22fe1700 1 --2- 172.21.15.23:0/1227249038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea1c0f5580 0x7fea1c105d20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fea10000c00 tx=0x7fea1000b040).stop 2021-11-19T08:43:16.727 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.741+0000 7fea22fe1700 1 -- 172.21.15.23:0/1227249038 shutdown_connections 2021-11-19T08:43:16.727 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.741+0000 7fea22fe1700 1 --2- 172.21.15.23:0/1227249038 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fea0c05c2a0 0x7fea0c05e760 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:16.728 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.741+0000 7fea22fe1700 1 --2- 172.21.15.23:0/1227249038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea1c0f5580 0x7fea1c105d20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:16.728 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.741+0000 7fea22fe1700 1 --2- 172.21.15.23:0/1227249038 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea1c0f6820 0x7fea1c106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:16.728 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.741+0000 7fea22fe1700 1 -- 172.21.15.23:0/1227249038 >> 172.21.15.23:0/1227249038 conn(0x7fea1c0f0ce0 msgr2=0x7fea1c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:16.728 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.741+0000 7fea22fe1700 1 -- 172.21.15.23:0/1227249038 shutdown_connections 2021-11-19T08:43:16.728 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:16.741+0000 7fea22fe1700 1 -- 172.21.15.23:0/1227249038 wait complete. 2021-11-19T08:43:16.729 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 58 2021-11-19T08:43:16.736 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:16 smithi023 conmon[33490]: cluster 2021-11-19T08:43:14. 2021-11-19T08:43:16.737 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:16 smithi023 conmon[33490]: 871717+0000 mgr.smithi023.ehlpcl (mgr.14164) 277 : cluster [DBG] pgmap v247: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:43:16.738 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:04.533906+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:16.739 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:43:17.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:16 smithi134 conmon[36363]: cluster 2021-11-19T08:43:14.871717+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:43:17.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:16 smithi134 conmon[36363]: 277 : cluster [DBG] pgmap v247: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:43:17.740 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:17.876 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.889+0000 7fadd6458700 1 -- 172.21.15.23:0/4184134002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadd00f5fe0 msgr2=0x7fadd00f6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:17.881 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.889+0000 7fadd6458700 1 --2- 172.21.15.23:0/4184134002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadd00f5fe0 0x7fadd00f6440 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fadc4004660 tx=0x7fadc4009b30).stop 2021-11-19T08:43:17.881 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.889+0000 7fadd6458700 1 -- 172.21.15.23:0/4184134002 shutdown_connections 2021-11-19T08:43:17.881 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.889+0000 7fadd6458700 1 --2- 172.21.15.23:0/4184134002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadd00f5fe0 0x7fadd00f6440 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:17.881 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.889+0000 7fadd6458700 1 --2- 172.21.15.23:0/4184134002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fadd00f4d40 0x7fadd00f5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:17.882 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.889+0000 7fadd6458700 1 -- 172.21.15.23:0/4184134002 >> 172.21.15.23:0/4184134002 conn(0x7fadd00f04a0 msgr2=0x7fadd00f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:17.882 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.890+0000 7fadd6458700 1 -- 172.21.15.23:0/4184134002 shutdown_connections 2021-11-19T08:43:17.882 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.890+0000 7fadd6458700 1 -- 172.21.15.23:0/4184134002 wait complete. 2021-11-19T08:43:17.883 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.890+0000 7fadd6458700 1 Processor -- start 2021-11-19T08:43:17.883 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.890+0000 7fadd6458700 1 -- start start 2021-11-19T08:43:17.883 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.890+0000 7fadd6458700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadd00f4d40 0x7fadd0106e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:17.883 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.890+0000 7fadd6458700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fadd00f5fe0 0x7fadd0107360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:17.884 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.890+0000 7fadd6458700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd0107910 con 0x7fadd00f4d40 2021-11-19T08:43:17.884 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.890+0000 7fadd6458700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd0107a50 con 0x7fadd00f5fe0 2021-11-19T08:43:17.884 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.891+0000 7fadcffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadd00f4d40 0x7fadd0106e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:17.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.891+0000 7fadcffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadd00f4d40 0x7fadd0106e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47812/0 (socket says 172.21.15.23:47812) 2021-11-19T08:43:17.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.891+0000 7fadcffff700 1 -- 172.21.15.23:0/3004687279 learned_addr learned my addr 172.21.15.23:0/3004687279 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:17.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.891+0000 7fadcf7fe700 1 --2- 172.21.15.23:0/3004687279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fadd00f5fe0 0x7fadd0107360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:17.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.891+0000 7fadcffff700 1 -- 172.21.15.23:0/3004687279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fadd00f5fe0 msgr2=0x7fadd0107360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:17.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.891+0000 7fadcffff700 1 --2- 172.21.15.23:0/3004687279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fadd00f5fe0 0x7fadd0107360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:17.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.891+0000 7fadcffff700 1 -- 172.21.15.23:0/3004687279 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc4005040 con 0x7fadd00f4d40 2021-11-19T08:43:17.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.891+0000 7fadcffff700 1 --2- 172.21.15.23:0/3004687279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadd00f4d40 0x7fadd0106e70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fadc000c530 tx=0x7fadc0007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:17.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.891+0000 7fadccff9700 1 -- 172.21.15.23:0/3004687279 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadc000f650 con 0x7fadd00f4d40 2021-11-19T08:43:17.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.891+0000 7fadccff9700 1 -- 172.21.15.23:0/3004687279 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fadc000fc60 con 0x7fadd00f4d40 2021-11-19T08:43:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.891+0000 7fadd6458700 1 -- 172.21.15.23:0/3004687279 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd010c4f0 con 0x7fadd00f4d40 2021-11-19T08:43:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.891+0000 7fadccff9700 1 -- 172.21.15.23:0/3004687279 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadc0012c60 con 0x7fadd00f4d40 2021-11-19T08:43:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.891+0000 7fadd6458700 1 -- 172.21.15.23:0/3004687279 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd010cab0 con 0x7fadd00f4d40 2021-11-19T08:43:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.892+0000 7fadd6458700 1 -- 172.21.15.23:0/3004687279 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd0027a00 con 0x7fadd00f4d40 2021-11-19T08:43:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.893+0000 7fadccff9700 1 -- 172.21.15.23:0/3004687279 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fadc0019030 con 0x7fadd00f4d40 2021-11-19T08:43:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.893+0000 7fadccff9700 1 --2- 172.21.15.23:0/3004687279 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fadb805c2a0 0x7fadb805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.893+0000 7fadccff9700 1 -- 172.21.15.23:0/3004687279 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fadc0075d70 con 0x7fadd00f4d40 2021-11-19T08:43:17.893 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.893+0000 7fadcf7fe700 1 --2- 172.21.15.23:0/3004687279 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fadb805c2a0 0x7fadb805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:17.893 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.893+0000 7fadcf7fe700 1 --2- 172.21.15.23:0/3004687279 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fadb805c2a0 0x7fadb805e760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fadc4004630 tx=0x7fadc4009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:17.893 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:17.897+0000 7fadccff9700 1 -- 172.21.15.23:0/3004687279 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fadc00462b0 con 0x7fadd00f4d40 2021-11-19T08:43:17.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:17 smithi023 conmon[33490]: audit 2021-11-19T08:43:16.740423+0000 mon.smithi023 (mon.0) 708 : audit [DBG] from='client.? 172.21.15.23:0/1227249038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:18.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:17 smithi134 conmon[36363]: audit 2021-11-19T08:43:16.740423+0000 mon.smithi023 (mon.0) 708 : audit [DBG] from='client.? 172.21.15.23:0/1227249038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:18.079 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:18.094+0000 7fadd6458700 1 -- 172.21.15.23:0/3004687279 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadd00fd4d0 con 0x7fadd00f4d40 2021-11-19T08:43:18.080 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:18.095+0000 7fadccff9700 1 -- 172.21.15.23:0/3004687279 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+2948 (secure 0 0 0) 0x7fadc0017020 con 0x7fadd00f4d40 2021-11-19T08:43:18.086 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:18.086 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":44},{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":57,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:04.533906+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":14578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14578":{"gid":14578,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":50,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6829/3057422404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3057422404},{"type":"v1","addr":"172.21.15.23:6829","nonce":3057422404}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:18.087 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:18.096+0000 7fadd6458700 1 -- 172.21.15.23:0/3004687279 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fadb805c2a0 msgr2=0x7fadb805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:18.088 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:18.096+0000 7fadd6458700 1 --2- 172.21.15.23:0/3004687279 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fadb805c2a0 0x7fadb805e760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fadc4004630 tx=0x7fadc4009b30).stop 2021-11-19T08:43:18.088 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:18.096+0000 7fadd6458700 1 -- 172.21.15.23:0/3004687279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadd00f4d40 msgr2=0x7fadd0106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:18.088 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:18.096+0000 7fadd6458700 1 --2- 172.21.15.23:0/3004687279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadd00f4d40 0x7fadd0106e70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fadc000c530 tx=0x7fadc0007e10).stop 2021-11-19T08:43:18.088 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:18.096+0000 7fadd6458700 1 -- 172.21.15.23:0/3004687279 shutdown_connections 2021-11-19T08:43:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:18.096+0000 7fadd6458700 1 --2- 172.21.15.23:0/3004687279 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fadb805c2a0 0x7fadb805e760 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:18.096+0000 7fadd6458700 1 --2- 172.21.15.23:0/3004687279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadd00f4d40 0x7fadd0106e70 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:18.096+0000 7fadd6458700 1 --2- 172.21.15.23:0/3004687279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fadd00f5fe0 0x7fadd0107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:18.097+0000 7fadd6458700 1 -- 172.21.15.23:0/3004687279 >> 172.21.15.23:0/3004687279 conn(0x7fadd00f04a0 msgr2=0x7fadd00fa520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:18.090 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:18.097+0000 7fadd6458700 1 -- 172.21.15.23:0/3004687279 shutdown_connections 2021-11-19T08:43:18.090 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:18.097+0000 7fadd6458700 1 -- 172.21.15.23:0/3004687279 wait complete. 2021-11-19T08:43:18.090 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 58 2021-11-19T08:43:18.093 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:04.533906+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 50, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 14578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14578': {'gid': 14578, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6829/3057422404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3057422404}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3057422404}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:18.093 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14578:up:rejoin 2021-11-19T08:43:18.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:18 smithi023 conmon[33490]: cluster 2021-11-19T08:43:16.872463+0000 mgr.smithi023.ehlpcl (mgr.14164) 278 : cluster [DBG] pgmap v248: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-19T08:43:18.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:18 smithi023 conmon[33490]: audit 2021-11-19T08:43:18.096040+0000 mon.smithi023 (mon.0) 709 : audit [DBG] from='client.? 172.21.15.23:0/3004687279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:19.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:18 smithi134 conmon[36363]: cluster 2021-11-19T08:43:16.872463+0000 mgr.smithi023.ehlpcl (mgr.14164) 278 2021-11-19T08:43:19.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:18 smithi134 conmon[36363]: : cluster [DBG] pgmap v248: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-19T08:43:19.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:18 smithi134 conmon[36363]: audit 2021-11-19T08:43:18.096040+0000 mon.smithi023 (mon.0) 709 : audit [DBG] from='client.? 172.21.15.23:0/3004687279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:19.095 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:19.233 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.246+0000 7f9e4dd92700 1 -- 172.21.15.23:0/2191060380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480f54d0 msgr2=0x7f9e480f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:19.234 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.246+0000 7f9e4dd92700 1 --2- 172.21.15.23:0/2191060380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480f54d0 0x7f9e480f58f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f9e3c004660 tx=0x7f9e3c009b30).stop 2021-11-19T08:43:19.234 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.247+0000 7f9e4dd92700 1 -- 172.21.15.23:0/2191060380 shutdown_connections 2021-11-19T08:43:19.234 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.247+0000 7f9e4dd92700 1 --2- 172.21.15.23:0/2191060380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e480f6770 0x7f9e480f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:19.235 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.247+0000 7f9e4dd92700 1 --2- 172.21.15.23:0/2191060380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480f54d0 0x7f9e480f58f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:19.235 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.247+0000 7f9e4dd92700 1 -- 172.21.15.23:0/2191060380 >> 172.21.15.23:0/2191060380 conn(0x7f9e480f0c10 msgr2=0x7f9e480f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:19.235 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.247+0000 7f9e4dd92700 1 -- 172.21.15.23:0/2191060380 shutdown_connections 2021-11-19T08:43:19.235 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.247+0000 7f9e4dd92700 1 -- 172.21.15.23:0/2191060380 wait complete. 2021-11-19T08:43:19.236 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.247+0000 7f9e4dd92700 1 Processor -- start 2021-11-19T08:43:19.236 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.247+0000 7f9e4dd92700 1 -- start start 2021-11-19T08:43:19.236 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.247+0000 7f9e4dd92700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480f54d0 0x7f9e48106da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:19.236 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e4dd92700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e480f6770 0x7f9e48107290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:19.237 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e4dd92700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e48107890 con 0x7f9e480f54d0 2021-11-19T08:43:19.237 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e4dd92700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e481080d0 con 0x7f9e480f6770 2021-11-19T08:43:19.237 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e4cd90700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480f54d0 0x7f9e48106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:19.237 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e4cd90700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480f54d0 0x7f9e48106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47820/0 (socket says 172.21.15.23:47820) 2021-11-19T08:43:19.238 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e4cd90700 1 -- 172.21.15.23:0/95760815 learned_addr learned my addr 172.21.15.23:0/95760815 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:19.238 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e47fff700 1 --2- 172.21.15.23:0/95760815 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e480f6770 0x7f9e48107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:19.238 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e4cd90700 1 -- 172.21.15.23:0/95760815 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e480f6770 msgr2=0x7f9e48107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:19.238 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e4cd90700 1 --2- 172.21.15.23:0/95760815 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e480f6770 0x7f9e48107290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:19.246 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e4cd90700 1 -- 172.21.15.23:0/95760815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3c005040 con 0x7f9e480f54d0 2021-11-19T08:43:19.247 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e47fff700 1 --2- 172.21.15.23:0/95760815 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e480f6770 0x7f9e48107290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:43:19.248 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e4cd90700 1 --2- 172.21.15.23:0/95760815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480f54d0 0x7f9e48106da0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f9e3c004e30 tx=0x7f9e3c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:19.248 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e457fa700 1 -- 172.21.15.23:0/95760815 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e3c018070 con 0x7f9e480f54d0 2021-11-19T08:43:19.248 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e457fa700 1 -- 172.21.15.23:0/95760815 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e3c013410 con 0x7f9e480f54d0 2021-11-19T08:43:19.248 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e457fa700 1 -- 172.21.15.23:0/95760815 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e3c01c3c0 con 0x7f9e480f54d0 2021-11-19T08:43:19.249 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e4dd92700 1 -- 172.21.15.23:0/95760815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e48108350 con 0x7f9e480f54d0 2021-11-19T08:43:19.249 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.248+0000 7f9e4dd92700 1 -- 172.21.15.23:0/95760815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e48108830 con 0x7f9e480f54d0 2021-11-19T08:43:19.249 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.249+0000 7f9e4dd92700 1 -- 172.21.15.23:0/95760815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e480f48c0 con 0x7f9e480f54d0 2021-11-19T08:43:19.251 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.250+0000 7f9e457fa700 1 -- 172.21.15.23:0/95760815 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9e3c01d420 con 0x7f9e480f54d0 2021-11-19T08:43:19.251 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.250+0000 7f9e457fa700 1 --2- 172.21.15.23:0/95760815 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f9e3805c300 0x7f9e3805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:19.252 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.250+0000 7f9e457fa700 1 -- 172.21.15.23:0/95760815 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f9e3c077e20 con 0x7f9e480f54d0 2021-11-19T08:43:19.252 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.250+0000 7f9e47fff700 1 --2- 172.21.15.23:0/95760815 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f9e3805c300 0x7f9e3805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:19.252 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.251+0000 7f9e47fff700 1 --2- 172.21.15.23:0/95760815 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f9e3805c300 0x7f9e3805e7c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f9e48106b30 tx=0x7f9e34009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:19.252 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.254+0000 7f9e457fa700 1 -- 172.21.15.23:0/95760815 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e3c04a780 con 0x7f9e480f54d0 2021-11-19T08:43:19.433 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.448+0000 7f9e4dd92700 1 -- 172.21.15.23:0/95760815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e4810d310 con 0x7f9e480f54d0 2021-11-19T08:43:19.434 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.449+0000 7f9e457fa700 1 -- 172.21.15.23:0/95760815 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+2573 (secure 0 0 0) 0x7f9e3c04dda0 con 0x7f9e480f54d0 2021-11-19T08:43:19.434 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:19.435 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":59,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:19.213275+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:replay","state_seq":3,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:19.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.450+0000 7f9e4dd92700 1 -- 172.21.15.23:0/95760815 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f9e3805c300 msgr2=0x7f9e3805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:19.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.450+0000 7f9e4dd92700 1 --2- 172.21.15.23:0/95760815 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f9e3805c300 0x7f9e3805e7c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f9e48106b30 tx=0x7f9e34009be0).stop 2021-11-19T08:43:19.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.450+0000 7f9e4dd92700 1 -- 172.21.15.23:0/95760815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480f54d0 msgr2=0x7f9e48106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:19.437 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.450+0000 7f9e4dd92700 1 --2- 172.21.15.23:0/95760815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480f54d0 0x7f9e48106da0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f9e3c004e30 tx=0x7f9e3c009b30).stop 2021-11-19T08:43:19.437 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.451+0000 7f9e4dd92700 1 -- 172.21.15.23:0/95760815 shutdown_connections 2021-11-19T08:43:19.437 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.451+0000 7f9e4dd92700 1 --2- 172.21.15.23:0/95760815 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f9e3805c300 0x7f9e3805e7c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:19.446 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.451+0000 7f9e4dd92700 1 --2- 172.21.15.23:0/95760815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480f54d0 0x7f9e48106da0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:19.447 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.451+0000 7f9e4dd92700 1 --2- 172.21.15.23:0/95760815 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e480f6770 0x7f9e48107290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:19.447 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.451+0000 7f9e4dd92700 1 -- 172.21.15.23:0/95760815 >> 172.21.15.23:0/95760815 conn(0x7f9e480f0c10 msgr2=0x7f9e4805ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:19.448 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.451+0000 7f9e4dd92700 1 -- 172.21.15.23:0/95760815 shutdown_connections 2021-11-19T08:43:19.448 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:19.451+0000 7f9e4dd92700 1 -- 172.21.15.23:0/95760815 wait complete. 2021-11-19T08:43:19.448 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 59 2021-11-19T08:43:19.449 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 59, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:19.213275+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:19.450 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14582:up:resolve 2021-11-19T08:43:19.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:19 smithi023 conmon[33490]: cluster 2021-11-19T08:43:19.212683+0000 mon.smithi023 (mon.0) 710 2021-11-19T08:43:19.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:19 smithi023 conmon[33490]: : cluster [WRN] Replacing daemon mds.cephfs.smithi023.wazanl as rank 1 with standby daemon mds.cephfs.smithi134.opxllb 2021-11-19T08:43:19.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:19 smithi023 conmon[33490]: cluster 2021-11-19T08:43:19.212737+0000 mon.smithi023 (mon.0) 711 : cluster [INF] MDS daemon mds.cephfs.smithi023.wazanl is removed because it is dead or otherwise unavailable. 2021-11-19T08:43:19.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:19 smithi023 conmon[33490]: cluster 2021-11-19T08:43:19.217857+0000 mon.smithi023 (mon.0 2021-11-19T08:43:19.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:19 smithi023 conmon[33490]: ) 712 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2021-11-19T08:43:19.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:19 smithi023 conmon[33490]: cluster 2021-11 2021-11-19T08:43:19.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:19 smithi023 conmon[33490]: -19T08:43:19.223727+0000 mon.smithi023 (mon. 2021-11-19T08:43:19.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:19 smithi023 conmon[33490]: 0) 713 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi134.opxllb=up:replay} 1 up:standby 2021-11-19T08:43:19.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:19 smithi023 conmon[33490]: audit 2021- 2021-11-19T08:43:19.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:19 smithi023 conmon[33490]: 11-19T08:43:19.450079+0000 mon.smithi023 ( 2021-11-19T08:43:19.897 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:19 smithi023 conmon[33490]: mon.0) 714 : audit [DBG] from='client.? 172.21.15.23:0/95760815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:20.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:19 smithi134 conmon[36363]: cluster 2021-11-19T08:43:19.212683+0000 mon.smithi023 (mon.0) 710 : cluster [WRN] Replacing daemon mds.cephfs.smithi023.wazanl as rank 1 with standby daemon mds.cephfs.smithi134.opxllb 2021-11-19T08:43:20.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:19 smithi134 conmon[36363]: cluster 2021-11-19T08:43:19.212737+0000 mon.smithi023 (mon.0) 711 : cluster [INF] MDS daemon mds.cephfs.smithi023.wazanl is removed because it is dead or otherwise unavailable. 2021-11-19T08:43:20.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:19 smithi134 conmon[36363]: cluster 2021-11-19T08:43:19.217857+0000 mon.smithi023 (mon.0) 712 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2021-11-19T08:43:20.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:19 smithi134 conmon[36363]: cluster 2021-11-19T08:43:20.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:19 smithi134 conmon[36363]: 2021-11-19T08:43:19.223727+0000 mon.smithi023 (mon.0) 713 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi134.opxllb=up:replay} 1 up:standby 2021-11-19T08:43:20.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:19 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:43:20.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:19 smithi134 conmon[36363]: 11-19T08:43:19.450079+0000 mon.smithi023 2021-11-19T08:43:20.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:19 smithi134 conmon[36363]: (mon.0) 714 : audit [DBG] from='client.? 172.21.15.23:0/95760815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:20.451 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:20.586 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.599+0000 7fd94b59e700 1 -- 172.21.15.23:0/1251474532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd94c0f54d0 msgr2=0x7fd94c0f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:20.587 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.599+0000 7fd94b59e700 1 --2- 172.21.15.23:0/1251474532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd94c0f54d0 0x7fd94c0f58f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fd93c004660 tx=0x7fd93c009b30).stop 2021-11-19T08:43:20.587 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.599+0000 7fd94b59e700 1 -- 172.21.15.23:0/1251474532 shutdown_connections 2021-11-19T08:43:20.587 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.599+0000 7fd94b59e700 1 --2- 172.21.15.23:0/1251474532 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd94c0f6770 0x7fd94c0f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:20.587 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.599+0000 7fd94b59e700 1 --2- 172.21.15.23:0/1251474532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd94c0f54d0 0x7fd94c0f58f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:20.588 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.599+0000 7fd94b59e700 1 -- 172.21.15.23:0/1251474532 >> 172.21.15.23:0/1251474532 conn(0x7fd94c0f0c10 msgr2=0x7fd94c0f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:20.588 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.600+0000 7fd94b59e700 1 -- 172.21.15.23:0/1251474532 shutdown_connections 2021-11-19T08:43:20.588 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.600+0000 7fd94b59e700 1 -- 172.21.15.23:0/1251474532 wait complete. 2021-11-19T08:43:20.588 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.600+0000 7fd94b59e700 1 Processor -- start 2021-11-19T08:43:20.589 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.600+0000 7fd94b59e700 1 -- start start 2021-11-19T08:43:20.589 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.600+0000 7fd94b59e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd94c0f54d0 0x7fd94c106da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:20.589 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.600+0000 7fd94b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd94c0f6770 0x7fd94c107290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:20.589 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.600+0000 7fd94b59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c107890 con 0x7fd94c0f6770 2021-11-19T08:43:20.589 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.600+0000 7fd94b59e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c1080d0 con 0x7fd94c0f54d0 2021-11-19T08:43:20.590 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.600+0000 7fd949d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd94c0f6770 0x7fd94c107290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:20.590 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.600+0000 7fd949d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd94c0f6770 0x7fd94c107290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47834/0 (socket says 172.21.15.23:47834) 2021-11-19T08:43:20.590 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.600+0000 7fd949d9b700 1 -- 172.21.15.23:0/3610996267 learned_addr learned my addr 172.21.15.23:0/3610996267 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:20.590 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.601+0000 7fd94a59c700 1 --2- 172.21.15.23:0/3610996267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd94c0f54d0 0x7fd94c106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:20.591 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.601+0000 7fd949d9b700 1 -- 172.21.15.23:0/3610996267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd94c0f54d0 msgr2=0x7fd94c106da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:20.591 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.601+0000 7fd949d9b700 1 --2- 172.21.15.23:0/3610996267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd94c0f54d0 0x7fd94c106da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:20.591 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.601+0000 7fd949d9b700 1 -- 172.21.15.23:0/3610996267 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c005040 con 0x7fd94c0f6770 2021-11-19T08:43:20.591 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.601+0000 7fd94a59c700 1 --2- 172.21.15.23:0/3610996267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd94c0f54d0 0x7fd94c106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:43:20.592 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.601+0000 7fd949d9b700 1 --2- 172.21.15.23:0/3610996267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd94c0f6770 0x7fd94c107290 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fd9400026d0 tx=0x7fd940009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:20.592 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.601+0000 7fd93affd700 1 -- 172.21.15.23:0/3610996267 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd940009320 con 0x7fd94c0f6770 2021-11-19T08:43:20.592 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.601+0000 7fd93affd700 1 -- 172.21.15.23:0/3610996267 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd94000f040 con 0x7fd94c0f6770 2021-11-19T08:43:20.592 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.601+0000 7fd94b59e700 1 -- 172.21.15.23:0/3610996267 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd94c1083b0 con 0x7fd94c0f6770 2021-11-19T08:43:20.592 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.601+0000 7fd93affd700 1 -- 172.21.15.23:0/3610996267 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9400147e0 con 0x7fd94c0f6770 2021-11-19T08:43:20.593 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.601+0000 7fd94b59e700 1 -- 172.21.15.23:0/3610996267 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd94c1088f0 con 0x7fd94c0f6770 2021-11-19T08:43:20.593 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.602+0000 7fd94b59e700 1 -- 172.21.15.23:0/3610996267 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd94c0f48c0 con 0x7fd94c0f6770 2021-11-19T08:43:20.594 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.602+0000 7fd93affd700 1 -- 172.21.15.23:0/3610996267 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd940009480 con 0x7fd94c0f6770 2021-11-19T08:43:20.594 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.603+0000 7fd93affd700 1 --2- 172.21.15.23:0/3610996267 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd93405c300 0x7fd93405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:20.595 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.603+0000 7fd93affd700 1 -- 172.21.15.23:0/3610996267 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fd940076910 con 0x7fd94c0f6770 2021-11-19T08:43:20.595 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.603+0000 7fd94a59c700 1 --2- 172.21.15.23:0/3610996267 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd93405c300 0x7fd93405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:20.595 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.603+0000 7fd94a59c700 1 --2- 172.21.15.23:0/3610996267 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd93405c300 0x7fd93405e7c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd94c107b60 tx=0x7fd93c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:20.596 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.607+0000 7fd93affd700 1 -- 172.21.15.23:0/3610996267 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd940049270 con 0x7fd94c0f6770 2021-11-19T08:43:20.787 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.803+0000 7fd94b59e700 1 -- 172.21.15.23:0/3610996267 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd94c10d310 con 0x7fd94c0f6770 2021-11-19T08:43:20.788 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.803+0000 7fd93affd700 1 -- 172.21.15.23:0/3610996267 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+2575 (secure 0 0 0) 0x7fd94004c890 con 0x7fd94c0f6770 2021-11-19T08:43:20.789 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:20.789 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:20.221556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:resolve","state_seq":18,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:20.791 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.805+0000 7fd94b59e700 1 -- 172.21.15.23:0/3610996267 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd93405c300 msgr2=0x7fd93405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:20.791 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.805+0000 7fd94b59e700 1 --2- 172.21.15.23:0/3610996267 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd93405c300 0x7fd93405e7c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd94c107b60 tx=0x7fd93c009b30).stop 2021-11-19T08:43:20.791 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.805+0000 7fd94b59e700 1 -- 172.21.15.23:0/3610996267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd94c0f6770 msgr2=0x7fd94c107290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:20.791 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.805+0000 7fd94b59e700 1 --2- 172.21.15.23:0/3610996267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd94c0f6770 0x7fd94c107290 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fd9400026d0 tx=0x7fd940009de0).stop 2021-11-19T08:43:20.792 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.805+0000 7fd94b59e700 1 -- 172.21.15.23:0/3610996267 shutdown_connections 2021-11-19T08:43:20.792 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.805+0000 7fd94b59e700 1 --2- 172.21.15.23:0/3610996267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd94c0f54d0 0x7fd94c106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:20.792 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.805+0000 7fd94b59e700 1 --2- 172.21.15.23:0/3610996267 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd93405c300 0x7fd93405e7c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:20.793 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.805+0000 7fd94b59e700 1 --2- 172.21.15.23:0/3610996267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd94c0f6770 0x7fd94c107290 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:20.793 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.805+0000 7fd94b59e700 1 -- 172.21.15.23:0/3610996267 >> 172.21.15.23:0/3610996267 conn(0x7fd94c0f0c10 msgr2=0x7fd94c05f760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:20.793 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.805+0000 7fd94b59e700 1 -- 172.21.15.23:0/3610996267 shutdown_connections 2021-11-19T08:43:20.793 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:20.805+0000 7fd94b59e700 1 -- 172.21.15.23:0/3610996267 wait complete. 2021-11-19T08:43:20.794 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 60 2021-11-19T08:43:20.800 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:20 smithi023 conmon[33490]: cluster 2021-11-19T08:43:18.873142+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:43:20.801 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:20 smithi023 conmon[33490]: .14164) 279 : cluster [DBG] pgmap v249: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:43:20.801 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:20 smithi023 conmon[33490]: cluster 2021-11-19T08:43:20.226892+0000 mon.smithi023 (mon.0) 715 : cluster [DBG] mds.? [v2:172.21.15.134:6826/348816881,v1:172.21.15.134:6827/348816881] up:resolve 2021-11-19T08:43:20.801 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:20 smithi023 conmon[33490]: cluster 2021-11-19T08:43:20.226966+0000 2021-11-19T08:43:20.801 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:20 smithi023 conmon[33490]: mon.smithi023 (mon.0) 716 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi134.opxllb=up:resolve} 1 up:standby 2021-11-19T08:43:20.803 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:20.221556+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:resolve', 'state_seq': 18, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:20.804 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14582:up:resolve 2021-11-19T08:43:21.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:20 smithi134 conmon[36363]: cluster 2021-11-19T08:43:18.873142+0000 mgr.smithi023.ehlpcl (mgr.14164) 279 : cluster 2021-11-19T08:43:21.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:20 smithi134 conmon[36363]: [DBG] pgmap v249: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:43:21.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:20 smithi134 conmon[36363]: cluster 2021-11-19T08:43:20.226892+0000 mon.smithi023 (mon.0) 715 : cluster [DBG] mds.? [v2:172.21.15.134:6826/348816881,v1:172.21.15.134:6827/348816881] up:resolve 2021-11-19T08:43:21.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:20 smithi134 conmon[36363]: cluster 2021-11-19T08:43:20 2021-11-19T08:43:21.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:20 smithi134 conmon[36363]: .226966+0000 mon.smithi023 (mon.0) 716 : cluster 2021-11-19T08:43:21.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:20 smithi134 conmon[36363]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi134.opxllb=up:resolve} 1 up:standby 2021-11-19T08:43:21.805 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:21.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: audit 2021-11-19T08:43:20.804462+0000 mon.smithi023 (mon.0) 717 : audit [DBG] from='client.? 172.21.15.23:0/3610996267' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:21.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: audit 2021-11-19T08:43:20.919612+0000 mon.smithi023 (mon.0) 718 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:43:21.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: audit 2021-11-19T08:43:21.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: 2021-11-19T08:43:20.920672+0000 mon.smithi023 (mon.0) 719 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:43:21.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: cluster 2021-11-19T08:43:21.228878+0000 mon.smithi023 (mon.0) 720 : cluster [DBG] 2021-11-19T08:43:21.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: mds.? [v2:172.21.15.23:6826/2135889102,v1:172.21.15.23:6827/2135889102] up:reconnect 2021-11-19T08:43:21.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: cluster 2021- 2021-11-19T08:43:21.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: 11-19T08:43:21.228943+0000 mon.smithi023 (mon.0) 721 : cluster 2021-11-19T08:43:21.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: [DBG] mds.? [v2:172.21.15.134:6826/348816881,v1:172.21.15.134:6827/348816881] up:reconnect 2021-11-19T08:43:21.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: cluster 2021-11-19T08: 2021-11-19T08:43:21.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: 43:21.229004+0000 mon.smithi023 (mon.0) 722 : 2021-11-19T08:43:21.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:reconnect,1=cephfs.smithi134.opxllb=up:reconnect} 1 up:standby 2021-11-19T08:43:21.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:43:21.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: 08:43:21.251895+0000 mds.cephfs.smithi023.fuvzkr (mds.0) 2021-11-19T08:43:21.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0239998 2021-11-19T08:43:21.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: cluster 2021-11-19T08:43:21.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: 2021-11-19T08:43:21.252541+0000 2021-11-19T08:43:21.896 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:21 smithi023 conmon[33490]: mds.cephfs.smithi134.opxllb (mds.1) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249994 2021-11-19T08:43:21.939 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.952+0000 7fbeaeb62700 1 -- 172.21.15.23:0/1412323792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbea80f7920 msgr2=0x7fbea80f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:21.940 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.952+0000 7fbeaeb62700 1 --2- 172.21.15.23:0/1412323792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbea80f7920 0x7fbea80f7d80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fbe94004660 tx=0x7fbe94009b30).stop 2021-11-19T08:43:21.940 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.953+0000 7fbeaeb62700 1 -- 172.21.15.23:0/1412323792 shutdown_connections 2021-11-19T08:43:21.940 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.953+0000 7fbeaeb62700 1 --2- 172.21.15.23:0/1412323792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbea80f7920 0x7fbea80f7d80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:21.941 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.953+0000 7fbeaeb62700 1 --2- 172.21.15.23:0/1412323792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbea80f6680 0x7fbea80f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:21.941 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.953+0000 7fbeaeb62700 1 -- 172.21.15.23:0/1412323792 >> 172.21.15.23:0/1412323792 conn(0x7fbea80f1de0 msgr2=0x7fbea80f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:21.941 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.953+0000 7fbeaeb62700 1 -- 172.21.15.23:0/1412323792 shutdown_connections 2021-11-19T08:43:21.941 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.953+0000 7fbeaeb62700 1 -- 172.21.15.23:0/1412323792 wait complete. 2021-11-19T08:43:21.942 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.953+0000 7fbeaeb62700 1 Processor -- start 2021-11-19T08:43:21.942 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.953+0000 7fbeaeb62700 1 -- start start 2021-11-19T08:43:21.942 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.953+0000 7fbeaeb62700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbea80f6680 0x7fbea8106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:21.942 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.953+0000 7fbeaeb62700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbea80f7920 0x7fbea8107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:21.942 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.953+0000 7fbeaeb62700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea8107930 con 0x7fbea80f7920 2021-11-19T08:43:21.943 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.953+0000 7fbeaeb62700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea8107a70 con 0x7fbea80f6680 2021-11-19T08:43:21.943 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.954+0000 7fbea7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbea80f7920 0x7fbea8107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:21.943 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.954+0000 7fbea7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbea80f7920 0x7fbea8107380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47848/0 (socket says 172.21.15.23:47848) 2021-11-19T08:43:21.943 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.954+0000 7fbea7fff700 1 -- 172.21.15.23:0/782197636 learned_addr learned my addr 172.21.15.23:0/782197636 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:21.944 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.954+0000 7fbeac8fe700 1 --2- 172.21.15.23:0/782197636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbea80f6680 0x7fbea8106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:21.944 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.954+0000 7fbea7fff700 1 -- 172.21.15.23:0/782197636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbea80f6680 msgr2=0x7fbea8106e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:21.944 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.954+0000 7fbea7fff700 1 --2- 172.21.15.23:0/782197636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbea80f6680 0x7fbea8106e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:21.944 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.954+0000 7fbea7fff700 1 -- 172.21.15.23:0/782197636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe94005040 con 0x7fbea80f7920 2021-11-19T08:43:21.946 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.954+0000 7fbea7fff700 1 --2- 172.21.15.23:0/782197636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbea80f7920 0x7fbea8107380 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fbe94009830 tx=0x7fbe94009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:21.946 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.954+0000 7fbea57fa700 1 -- 172.21.15.23:0/782197636 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe94018070 con 0x7fbea80f7920 2021-11-19T08:43:21.947 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.954+0000 7fbea57fa700 1 -- 172.21.15.23:0/782197636 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe94013410 con 0x7fbea80f7920 2021-11-19T08:43:21.947 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.954+0000 7fbeaeb62700 1 -- 172.21.15.23:0/782197636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea810c4b0 con 0x7fbea80f7920 2021-11-19T08:43:21.947 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.954+0000 7fbea57fa700 1 -- 172.21.15.23:0/782197636 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe9401c400 con 0x7fbea80f7920 2021-11-19T08:43:21.947 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.954+0000 7fbeaeb62700 1 -- 172.21.15.23:0/782197636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea810ca70 con 0x7fbea80f7920 2021-11-19T08:43:21.948 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.955+0000 7fbeaeb62700 1 -- 172.21.15.23:0/782197636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea8027a00 con 0x7fbea80f7920 2021-11-19T08:43:21.949 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.955+0000 7fbea57fa700 1 -- 172.21.15.23:0/782197636 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fbe9401d420 con 0x7fbea80f7920 2021-11-19T08:43:21.949 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.956+0000 7fbea57fa700 1 --2- 172.21.15.23:0/782197636 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbe9805c2a0 0x7fbe9805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:21.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.956+0000 7fbea57fa700 1 -- 172.21.15.23:0/782197636 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fbe940780a0 con 0x7fbea80f7920 2021-11-19T08:43:21.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.956+0000 7fbeac8fe700 1 --2- 172.21.15.23:0/782197636 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbe9805c2a0 0x7fbe9805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:21.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.956+0000 7fbeac8fe700 1 --2- 172.21.15.23:0/782197636 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbe9805c2a0 0x7fbe9805e760 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fbea80f7780 tx=0x7fbe9c009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:21.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:21.960+0000 7fbea57fa700 1 -- 172.21.15.23:0/782197636 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe9404aa00 con 0x7fbea80f7920 2021-11-19T08:43:22.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: audit 2021-11-19T08:43:20.804462+0000 mon.smithi023 (mon.0) 717 : audit [DBG] from='client.? 172.21.15.23:0/3610996267' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:22.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: audit 2021-11-19T08:43:20.919612+0000 mon.smithi023 (mon.0 2021-11-19T08:43:22.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: ) 718 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:43:22.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: audit 2021-11-19T08:43:20.920672+0000 mon.smithi023 (mon.0) 719 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:43:22.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: cluster 2021-11-19T08 2021-11-19T08:43:22.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: :43:21.228878+0000 mon.smithi023 (mon.0) 720 : cluster [DBG] mds.? [v2:172.21.15.23:6826/2135889102,v1:172.21.15.23:6827/2135889102] up:reconnect 2021-11-19T08:43:22.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: cluster 2021-11-19 2021-11-19T08:43:22.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: T08:43:21.228943+0000 mon.smithi023 2021-11-19T08:43:22.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: (mon.0) 721 : cluster [DBG] mds.? [v2:172.21.15.134:6826/348816881,v1:172.21.15.134:6827/348816881] up:reconnect 2021-11-19T08:43:22.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: cluster 2021-11-19T08:43:21.229004+0000 2021-11-19T08:43:22.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: mon.smithi023 (mon.0) 722 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:reconnect,1=cephfs.smithi134.opxllb=up:reconnect} 1 up:standby 2021-11-19T08:43:22.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: cluster 2021 2021-11-19T08:43:22.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: -11-19T08:43:21 2021-11-19T08:43:22.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: .251895+0000 mds.cephfs.smithi023.fuvzkr (mds.0) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0239998 2021-11-19T08:43:22.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: cluster 2021-11-19T08:43:22.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: 2021-11-19T08:43: 2021-11-19T08:43:22.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: 21.252541+0000 mds.cephfs.smithi134.opxllb (mds.1) 1 : 2021-11-19T08:43:22.044 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:21 smithi134 conmon[36363]: cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249994 2021-11-19T08:43:22.142 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:22.157+0000 7fbeaeb62700 1 -- 172.21.15.23:0/782197636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbea80fee10 con 0x7fbea80f7920 2021-11-19T08:43:22.142 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:22.158+0000 7fbea57fa700 1 -- 172.21.15.23:0/782197636 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2580 (secure 0 0 0) 0x7fbe9404e020 con 0x7fbea80f7920 2021-11-19T08:43:22.143 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:22.143 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":61,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:21.224147+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:22.147 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:22.159+0000 7fbeaeb62700 1 -- 172.21.15.23:0/782197636 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbe9805c2a0 msgr2=0x7fbe9805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:22.147 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:22.159+0000 7fbeaeb62700 1 --2- 172.21.15.23:0/782197636 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbe9805c2a0 0x7fbe9805e760 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fbea80f7780 tx=0x7fbe9c009040).stop 2021-11-19T08:43:22.147 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:22.159+0000 7fbeaeb62700 1 -- 172.21.15.23:0/782197636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbea80f7920 msgr2=0x7fbea8107380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:22.148 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:22.159+0000 7fbeaeb62700 1 --2- 172.21.15.23:0/782197636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbea80f7920 0x7fbea8107380 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fbe94009830 tx=0x7fbe94009b30).stop 2021-11-19T08:43:22.148 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:22.159+0000 7fbeaeb62700 1 -- 172.21.15.23:0/782197636 shutdown_connections 2021-11-19T08:43:22.149 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:22.160+0000 7fbeaeb62700 1 --2- 172.21.15.23:0/782197636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbea80f6680 0x7fbea8106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:22.149 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:22.160+0000 7fbeaeb62700 1 --2- 172.21.15.23:0/782197636 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fbe9805c2a0 0x7fbe9805e760 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:22.150 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:22.160+0000 7fbeaeb62700 1 --2- 172.21.15.23:0/782197636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbea80f7920 0x7fbea8107380 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:22.151 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:22.160+0000 7fbeaeb62700 1 -- 172.21.15.23:0/782197636 >> 172.21.15.23:0/782197636 conn(0x7fbea80f1de0 msgr2=0x7fbea80fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:22.151 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:22.160+0000 7fbeaeb62700 1 -- 172.21.15.23:0/782197636 shutdown_connections 2021-11-19T08:43:22.151 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:22.160+0000 7fbeaeb62700 1 -- 172.21.15.23:0/782197636 wait complete. 2021-11-19T08:43:22.152 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 61 2021-11-19T08:43:22.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:21.224147+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:reconnect', 'state_seq': 11, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:reconnect', 'state_seq': 19, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:22.157 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14582:up:reconnect 2021-11-19T08:43:22.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:22 smithi023 conmon[33490]: cluster 2021-11-19T08:43:20.873612+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:43:22.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:22 smithi023 conmon[33490]: mgr.14164) 280 : cluster [DBG] pgmap v251: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-19T08:43:22.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:22 smithi023 conmon[33490]: audit 2021-11-19T08:43:22.158878+0000 mon.smithi023 (mon.0) 723 : audit [DBG] from='client.? 172.21.15.23:0/782197636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:22.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:22 smithi023 conmon[33490]: cluster 2021-11-19T08:43:22.231468+0000 mon.smithi023 (mon.0) 724 : cluster [DBG] mds.? [v2:172.21.15.23:6826/2135889102,v1:172.21.15.23:6827/2135889102] up:rejoin 2021-11-19T08:43:22.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:22 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:43:22.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:22 smithi023 conmon[33490]: 08:43:22.231529+0000 mon.smithi023 (mon 2021-11-19T08:43:22.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:22 smithi023 conmon[33490]: .0) 725 : cluster [DBG] mds.? [v2:172.21.15.134:6826/348816881,v1:172.21.15.134:6827/348816881] up:rejoin 2021-11-19T08:43:22.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:22 smithi023 conmon[33490]: cluster 2021-11-19T08:43:22.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:22 smithi023 conmon[33490]: 2021-11-19T08:43:22.231591+0000 mon.smithi023 2021-11-19T08:43:22.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:22 smithi023 conmon[33490]: (mon.0) 726 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:rejoin,1=cephfs.smithi134.opxllb=up:rejoin} 1 up:standby 2021-11-19T08:43:22.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:22 smithi023 conmon[33490]: 2021-11-19T08:43:22.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:22 smithi023 conmon[33490]: cluster 2021-11-19T08:43:22 2021-11-19T08:43:22.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:22 smithi023 conmon[33490]: .233852+0000 mon.smithi023 (mon.0) 727 : cluster [INF] daemon mds.cephfs.smithi134.opxllb is now active in filesystem cephfs as rank 1 2021-11-19T08:43:22.895 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:22 smithi023 conmon[33490]: 2021-11-19T08:43:23.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:22 smithi134 conmon[36363]: cluster 2021-11-19T08:43:20.873612+0000 mgr.smithi023.ehlpcl (mgr.14164) 280 : cluster [DBG] pgmap v251: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-19T08:43:23.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:22 smithi134 conmon[36363]: 2021-11-19T08:43:23.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:22 smithi134 conmon[36363]: audit 2021-11-19T08:43:22.158878+0000 mon.smithi023 (mon.0) 723 : audit [DBG] from='client.? 172.21.15.23:0/782197636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:23.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:22 smithi134 conmon[36363]: cluster 2021-11-19T08:43: 2021-11-19T08:43:23.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:22 smithi134 conmon[36363]: 22.231468+0000 mon.smithi023 (mon.0) 724 : cluster [DBG] mds.? [v2:172.21.15.23:6826/2135889102,v1:172.21.15.23:6827/2135889102] up:rejoin 2021-11-19T08:43:23.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:22 smithi134 conmon[36363]: cluster 2021-11-19T08 2021-11-19T08:43:23.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:22 smithi134 conmon[36363]: :43:22.231529+0000 mon.smithi023 ( 2021-11-19T08:43:23.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:22 smithi134 conmon[36363]: mon.0) 725 : cluster [DBG] mds.? [v2:172.21.15.134:6826/348816881,v1:172.21.15.134:6827/348816881] up:rejoin 2021-11-19T08:43:23.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:22 smithi134 conmon[36363]: cluster 2021-11-19T 2021-11-19T08:43:23.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:22 smithi134 conmon[36363]: 08:43:22.231591+0000 mon.smithi023 (mon.0) 2021-11-19T08:43:23.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:22 smithi134 conmon[36363]: 726 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:rejoin,1=cephfs.smithi134.opxllb=up:rejoin} 1 up:standby 2021-11-19T08:43:23.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:22 smithi134 conmon[36363]: cluster 2021-11-19T 2021-11-19T08:43:23.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:22 smithi134 conmon[36363]: 08:43:22.233852+0000 mon.smithi023 (mon.0 2021-11-19T08:43:23.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:22 smithi134 conmon[36363]: ) 727 : cluster [INF] daemon mds.cephfs.smithi134.opxllb is now active in filesystem cephfs as rank 1 2021-11-19T08:43:23.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:23.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.303+0000 7fac5c2ed700 1 -- 172.21.15.23:0/1686522300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac540f7920 msgr2=0x7fac540f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:23.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.303+0000 7fac5c2ed700 1 --2- 172.21.15.23:0/1686522300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac540f7920 0x7fac540f7d80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fac44004660 tx=0x7fac44009b30).stop 2021-11-19T08:43:23.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.303+0000 7fac5c2ed700 1 -- 172.21.15.23:0/1686522300 shutdown_connections 2021-11-19T08:43:23.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.303+0000 7fac5c2ed700 1 --2- 172.21.15.23:0/1686522300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac540f7920 0x7fac540f7d80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:23.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.303+0000 7fac5c2ed700 1 --2- 172.21.15.23:0/1686522300 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac540f6680 0x7fac540f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:23.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.303+0000 7fac5c2ed700 1 -- 172.21.15.23:0/1686522300 >> 172.21.15.23:0/1686522300 conn(0x7fac540f1de0 msgr2=0x7fac540f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:23.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.303+0000 7fac5c2ed700 1 -- 172.21.15.23:0/1686522300 shutdown_connections 2021-11-19T08:43:23.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.303+0000 7fac5c2ed700 1 -- 172.21.15.23:0/1686522300 wait complete. 2021-11-19T08:43:23.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.303+0000 7fac5c2ed700 1 Processor -- start 2021-11-19T08:43:23.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.303+0000 7fac5c2ed700 1 -- start start 2021-11-19T08:43:23.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac5c2ed700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac540f6680 0x7fac5406b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:23.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac5c2ed700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac540f7920 0x7fac54069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:23.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac5c2ed700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5406bd70 con 0x7fac540f7920 2021-11-19T08:43:23.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac5c2ed700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5406a310 con 0x7fac540f6680 2021-11-19T08:43:23.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac59888700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac540f7920 0x7fac54069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:23.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac59888700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac540f7920 0x7fac54069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47862/0 (socket says 172.21.15.23:47862) 2021-11-19T08:43:23.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac59888700 1 -- 172.21.15.23:0/745126484 learned_addr learned my addr 172.21.15.23:0/745126484 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:23.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac5a089700 1 --2- 172.21.15.23:0/745126484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac540f6680 0x7fac5406b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:23.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac59888700 1 -- 172.21.15.23:0/745126484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac540f6680 msgr2=0x7fac5406b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:23.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac59888700 1 --2- 172.21.15.23:0/745126484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac540f6680 0x7fac5406b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:23.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac59888700 1 -- 172.21.15.23:0/745126484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac44005040 con 0x7fac540f7920 2021-11-19T08:43:23.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac5a089700 1 --2- 172.21.15.23:0/745126484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac540f6680 0x7fac5406b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:43:23.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac59888700 1 --2- 172.21.15.23:0/745126484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac540f7920 0x7fac54069e20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fac44007a40 tx=0x7fac44009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:23.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac4affd700 1 -- 172.21.15.23:0/745126484 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac44018070 con 0x7fac540f7920 2021-11-19T08:43:23.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac5c2ed700 1 -- 172.21.15.23:0/745126484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5406a530 con 0x7fac540f7920 2021-11-19T08:43:23.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac4affd700 1 -- 172.21.15.23:0/745126484 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac44013410 con 0x7fac540f7920 2021-11-19T08:43:23.298 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac4affd700 1 -- 172.21.15.23:0/745126484 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac4401c620 con 0x7fac540f7920 2021-11-19T08:43:23.298 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.304+0000 7fac5c2ed700 1 -- 172.21.15.23:0/745126484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5406aa90 con 0x7fac540f7920 2021-11-19T08:43:23.298 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.305+0000 7fac5c2ed700 1 -- 172.21.15.23:0/745126484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac54046330 con 0x7fac540f7920 2021-11-19T08:43:23.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.306+0000 7fac4affd700 1 -- 172.21.15.23:0/745126484 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fac4401c8e0 con 0x7fac540f7920 2021-11-19T08:43:23.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.306+0000 7fac4affd700 1 --2- 172.21.15.23:0/745126484 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fac4005c2a0 0x7fac4005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:23.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.306+0000 7fac4affd700 1 -- 172.21.15.23:0/745126484 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fac44078170 con 0x7fac540f7920 2021-11-19T08:43:23.300 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.306+0000 7fac5a089700 1 --2- 172.21.15.23:0/745126484 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fac4005c2a0 0x7fac4005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:23.300 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.306+0000 7fac5a089700 1 --2- 172.21.15.23:0/745126484 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fac4005c2a0 0x7fac4005e760 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fac540f7780 tx=0x7fac50009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:23.300 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.310+0000 7fac4affd700 1 -- 172.21.15.23:0/745126484 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac4404aa70 con 0x7fac540f7920 2021-11-19T08:43:23.491 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.506+0000 7fac5c2ed700 1 -- 172.21.15.23:0/745126484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac54027a00 con 0x7fac540f7920 2021-11-19T08:43:23.491 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.507+0000 7fac4affd700 1 -- 172.21.15.23:0/745126484 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2580 (secure 0 0 0) 0x7fac4404e090 con 0x7fac540f7920 2021-11-19T08:43:23.492 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:23.492 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:23.229193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:23.494 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.508+0000 7fac5c2ed700 1 -- 172.21.15.23:0/745126484 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fac4005c2a0 msgr2=0x7fac4005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:23.494 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.508+0000 7fac5c2ed700 1 --2- 172.21.15.23:0/745126484 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fac4005c2a0 0x7fac4005e760 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fac540f7780 tx=0x7fac50009040).stop 2021-11-19T08:43:23.494 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.508+0000 7fac5c2ed700 1 -- 172.21.15.23:0/745126484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac540f7920 msgr2=0x7fac54069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:23.494 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.509+0000 7fac5c2ed700 1 --2- 172.21.15.23:0/745126484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac540f7920 0x7fac54069e20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fac44007a40 tx=0x7fac44009b30).stop 2021-11-19T08:43:23.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.509+0000 7fac5c2ed700 1 -- 172.21.15.23:0/745126484 shutdown_connections 2021-11-19T08:43:23.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.509+0000 7fac5c2ed700 1 --2- 172.21.15.23:0/745126484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac540f6680 0x7fac5406b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:23.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.509+0000 7fac5c2ed700 1 --2- 172.21.15.23:0/745126484 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fac4005c2a0 0x7fac4005e760 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:23.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.509+0000 7fac5c2ed700 1 --2- 172.21.15.23:0/745126484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac540f7920 0x7fac54069e20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:23.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.509+0000 7fac5c2ed700 1 -- 172.21.15.23:0/745126484 >> 172.21.15.23:0/745126484 conn(0x7fac540f1de0 msgr2=0x7fac540fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:23.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.509+0000 7fac5c2ed700 1 -- 172.21.15.23:0/745126484 shutdown_connections 2021-11-19T08:43:23.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:23.509+0000 7fac5c2ed700 1 -- 172.21.15.23:0/745126484 wait complete. 2021-11-19T08:43:23.496 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 63 2021-11-19T08:43:23.513 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:23.229193+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:23.514 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14582:up:clientreplay 2021-11-19T08:43:24.515 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:24.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:24 smithi134 conmon[36363]: cluster 2021-11-19T08:43:22.874020+0000 mgr.smithi023.ehlpcl (mgr.14164) 281 : cluster [DBG] pgmap v252: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:43:24.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:24 smithi134 conmon[36363]: cluster 2021-11-19T08:43:23.233506+0000 mon.smithi023 (mon.0) 728 : cluster [DBG] mds.? [v2:172.21.15.23:6826/2135889102,v1:172.21.15.23:6827/2135889102] up:clientreplay 2021-11-19T08:43:24.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:24 smithi134 conmon[36363]: cluster 2021-11-19T08:43:23.233554+0000 mon.smithi023 (mon.0) 729 2021-11-19T08:43:24.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:24 smithi134 conmon[36363]: : cluster [DBG] mds.? [v2:172.21.15.134:6826/348816881,v1:172.21.15.134:6827/348816881] up:active 2021-11-19T08:43:24.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:24 smithi134 conmon[36363]: cluster 2021-11-19 2021-11-19T08:43:24.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:24 smithi134 conmon[36363]: T08:43:23.233606+0000 mon.smithi023 (mon.0) 730 : cluster [DBG] 2021-11-19T08:43:24.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:24 smithi134 conmon[36363]: fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:clientreplay,1=cephfs.smithi134.opxllb=up:active} 1 up:standby 2021-11-19T08:43:24.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:24 smithi134 conmon[36363]: audit 2021-11-19T08:43: 2021-11-19T08:43:24.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:24 smithi134 conmon[36363]: 23.508021+0000 mon.smithi023 (mon.0) 731 : 2021-11-19T08:43:24.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:24 smithi134 conmon[36363]: audit [DBG] from='client.? 172.21.15.23:0/745126484' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:24.543 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:24 smithi023 conmon[33490]: cluster 2021-11-19T08:43:22.874020+0000 mgr.smithi023.ehlpcl (mgr.14164) 281 : cluster [DBG] pgmap v252: 65 pgs: 65 active+clean; 4.7 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:43:24.544 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:24 smithi023 conmon[33490]: cluster 2021-11-19T08:43:23.233506+0000 mon.smithi023 (mon.0) 728 : cluster [DBG] mds.? [v2:172.21.15.23:6826/2135889102,v1:172.21.15.23:6827/2135889102] up:clientreplay 2021-11-19T08:43:24.544 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:24 smithi023 conmon[33490]: cluster 2021-11-19T08:43:23.233554+0000 mon.smithi023 (mon.0) 729 : cluster [DBG] mds.? [v2:172.21.15.134:6826/348816881,v1:172.21.15.134:6827/348816881] up:active 2021-11-19T08:43:24.544 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:24 smithi023 conmon[33490]: cluster 2021-11-19T08:43:23.233606+0000 mon.smithi023 (mon.0) 730 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:clientreplay,1=cephfs.smithi134.opxllb=up:active} 1 up:standby 2021-11-19T08:43:24.544 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:24 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:43:24.545 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:24 smithi023 conmon[33490]: -19T08:43:23.508021+0000 mon.smithi023 (mon.0) 731 : audit [DBG] 2021-11-19T08:43:24.545 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:24 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/745126484' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:24.649 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.663+0000 7f9acda54700 1 -- 172.21.15.23:0/2438649704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ac80f6790 msgr2=0x7f9ac80f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:24.649 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.663+0000 7f9acda54700 1 --2- 172.21.15.23:0/2438649704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ac80f6790 0x7f9ac80f6bf0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f9ab0004660 tx=0x7f9ab0009b30).stop 2021-11-19T08:43:24.650 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.663+0000 7f9acda54700 1 -- 172.21.15.23:0/2438649704 shutdown_connections 2021-11-19T08:43:24.650 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.663+0000 7f9acda54700 1 --2- 172.21.15.23:0/2438649704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ac80f6790 0x7f9ac80f6bf0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:24.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.663+0000 7f9acda54700 1 --2- 172.21.15.23:0/2438649704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ac80f54f0 0x7f9ac80f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:24.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.663+0000 7f9acda54700 1 -- 172.21.15.23:0/2438649704 >> 172.21.15.23:0/2438649704 conn(0x7f9ac80f0c30 msgr2=0x7f9ac80f3070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:24.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.663+0000 7f9acda54700 1 -- 172.21.15.23:0/2438649704 shutdown_connections 2021-11-19T08:43:24.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.663+0000 7f9acda54700 1 -- 172.21.15.23:0/2438649704 wait complete. 2021-11-19T08:43:24.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.663+0000 7f9acda54700 1 Processor -- start 2021-11-19T08:43:24.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.663+0000 7f9acda54700 1 -- start start 2021-11-19T08:43:24.654 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9acda54700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ac80f54f0 0x7f9ac80fd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:24.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9acda54700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ac80f6790 0x7f9ac80fbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:24.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9acda54700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac80fc050 con 0x7f9ac80f54f0 2021-11-19T08:43:24.656 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9acda54700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac80fc190 con 0x7f9ac80f6790 2021-11-19T08:43:24.657 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9ac6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ac80f54f0 0x7f9ac80fd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:24.657 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9ac6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ac80f54f0 0x7f9ac80fd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47872/0 (socket says 172.21.15.23:47872) 2021-11-19T08:43:24.658 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9ac6ffd700 1 -- 172.21.15.23:0/1670979527 learned_addr learned my addr 172.21.15.23:0/1670979527 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:24.658 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9abffff700 1 --2- 172.21.15.23:0/1670979527 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ac80f6790 0x7f9ac80fbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:24.659 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9ac6ffd700 1 -- 172.21.15.23:0/1670979527 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ac80f6790 msgr2=0x7f9ac80fbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:24.659 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9ac6ffd700 1 --2- 172.21.15.23:0/1670979527 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ac80f6790 0x7f9ac80fbaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:24.659 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9ac6ffd700 1 -- 172.21.15.23:0/1670979527 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab0005040 con 0x7f9ac80f54f0 2021-11-19T08:43:24.660 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9ac6ffd700 1 --2- 172.21.15.23:0/1670979527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ac80f54f0 0x7f9ac80fd670 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f9ab8002700 tx=0x7f9ab8009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:24.660 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9abf7fe700 1 -- 172.21.15.23:0/1670979527 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ab8009320 con 0x7f9ac80f54f0 2021-11-19T08:43:24.661 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9acda54700 1 -- 172.21.15.23:0/1670979527 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac80fc470 con 0x7f9ac80f54f0 2021-11-19T08:43:24.661 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9abf7fe700 1 -- 172.21.15.23:0/1670979527 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ab800f040 con 0x7f9ac80f54f0 2021-11-19T08:43:24.661 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9abf7fe700 1 -- 172.21.15.23:0/1670979527 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ab8014900 con 0x7f9ac80f54f0 2021-11-19T08:43:24.662 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.664+0000 7f9acda54700 1 -- 172.21.15.23:0/1670979527 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac810eda0 con 0x7f9ac80f54f0 2021-11-19T08:43:24.665 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.665+0000 7f9acda54700 1 -- 172.21.15.23:0/1670979527 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac8045ad0 con 0x7f9ac80f54f0 2021-11-19T08:43:24.665 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.666+0000 7f9abf7fe700 1 -- 172.21.15.23:0/1670979527 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9ab8009480 con 0x7f9ac80f54f0 2021-11-19T08:43:24.665 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.666+0000 7f9abf7fe700 1 --2- 172.21.15.23:0/1670979527 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f9ab405c300 0x7f9ab405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:24.666 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.666+0000 7f9abf7fe700 1 -- 172.21.15.23:0/1670979527 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f9ab8076570 con 0x7f9ac80f54f0 2021-11-19T08:43:24.666 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.666+0000 7f9abffff700 1 --2- 172.21.15.23:0/1670979527 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f9ab405c300 0x7f9ab405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:24.666 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.667+0000 7f9abffff700 1 --2- 172.21.15.23:0/1670979527 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f9ab405c300 0x7f9ab405e7c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f9ab0009d60 tx=0x7f9ab0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:24.666 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.670+0000 7f9abf7fe700 1 -- 172.21.15.23:0/1670979527 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ab8048ed0 con 0x7f9ac80f54f0 2021-11-19T08:43:24.851 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.867+0000 7f9acda54700 1 -- 172.21.15.23:0/1670979527 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ac8027a00 con 0x7f9ac80f54f0 2021-11-19T08:43:24.852 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.867+0000 7f9abf7fe700 1 -- 172.21.15.23:0/1670979527 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2580 (secure 0 0 0) 0x7f9ab804c4f0 con 0x7f9ac80f54f0 2021-11-19T08:43:24.852 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:24.853 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:23.229193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:24.854 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.869+0000 7f9acda54700 1 -- 172.21.15.23:0/1670979527 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f9ab405c300 msgr2=0x7f9ab405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:24.854 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.869+0000 7f9acda54700 1 --2- 172.21.15.23:0/1670979527 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f9ab405c300 0x7f9ab405e7c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f9ab0009d60 tx=0x7f9ab0009b30).stop 2021-11-19T08:43:24.855 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.869+0000 7f9acda54700 1 -- 172.21.15.23:0/1670979527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ac80f54f0 msgr2=0x7f9ac80fd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:24.855 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.869+0000 7f9acda54700 1 --2- 172.21.15.23:0/1670979527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ac80f54f0 0x7f9ac80fd670 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f9ab8002700 tx=0x7f9ab8009d80).stop 2021-11-19T08:43:24.855 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.869+0000 7f9acda54700 1 -- 172.21.15.23:0/1670979527 shutdown_connections 2021-11-19T08:43:24.855 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.869+0000 7f9acda54700 1 --2- 172.21.15.23:0/1670979527 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f9ab405c300 0x7f9ab405e7c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:24.856 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.869+0000 7f9acda54700 1 --2- 172.21.15.23:0/1670979527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ac80f54f0 0x7f9ac80fd670 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:24.856 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.869+0000 7f9acda54700 1 --2- 172.21.15.23:0/1670979527 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ac80f6790 0x7f9ac80fbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:24.856 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.869+0000 7f9acda54700 1 -- 172.21.15.23:0/1670979527 >> 172.21.15.23:0/1670979527 conn(0x7f9ac80f0c30 msgr2=0x7f9ac80fa1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:24.856 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.869+0000 7f9acda54700 1 -- 172.21.15.23:0/1670979527 shutdown_connections 2021-11-19T08:43:24.857 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:24.869+0000 7f9acda54700 1 -- 172.21.15.23:0/1670979527 wait complete. 2021-11-19T08:43:24.857 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 63 2021-11-19T08:43:24.866 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:23.229193+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:24.866 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14582:up:clientreplay 2021-11-19T08:43:25.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:25 smithi134 conmon[36363]: audit 2021-11-19T08:43:24.868414+0000 mon.smithi023 (mon.0) 732 : audit [DBG] from='client.? 172.21.15.23:0/1670979527' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:25.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:25 smithi023 conmon[33490]: audit 2021-11-19T08:43:24.868414+0000 mon.smithi023 (mon. 2021-11-19T08:43:25.834 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:25 smithi023 conmon[33490]: 0) 732 : audit [DBG] from='client.? 172.21.15.23:0/1670979527' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:25.868 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:26.010 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.017+0000 7f996e27b700 1 -- 172.21.15.23:0/3977442496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968060140 msgr2=0x7f99680fb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:26.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.017+0000 7f996e27b700 1 --2- 172.21.15.23:0/3977442496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968060140 0x7f99680fb540 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f9964004660 tx=0x7f9964009b30).stop 2021-11-19T08:43:26.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996e27b700 1 -- 172.21.15.23:0/3977442496 shutdown_connections 2021-11-19T08:43:26.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996e27b700 1 --2- 172.21.15.23:0/3977442496 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99680fba30 0x7f99680fde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:26.012 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996e27b700 1 --2- 172.21.15.23:0/3977442496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968060140 0x7f99680fb540 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:26.012 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996e27b700 1 -- 172.21.15.23:0/3977442496 >> 172.21.15.23:0/3977442496 conn(0x7f99680f0ba0 msgr2=0x7f99680f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:26.013 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996e27b700 1 -- 172.21.15.23:0/3977442496 shutdown_connections 2021-11-19T08:43:26.013 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996e27b700 1 -- 172.21.15.23:0/3977442496 wait complete. 2021-11-19T08:43:26.014 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996e27b700 1 Processor -- start 2021-11-19T08:43:26.014 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996e27b700 1 -- start start 2021-11-19T08:43:26.014 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996e27b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968060140 0x7f9968108f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:26.015 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996e27b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99680fba30 0x7f9968109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:26.015 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996e27b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99681099e0 con 0x7f9968060140 2021-11-19T08:43:26.016 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996e27b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9968109b20 con 0x7f99680fba30 2021-11-19T08:43:26.016 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996d279700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968060140 0x7f9968108f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:26.017 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996d279700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968060140 0x7f9968108f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47884/0 (socket says 172.21.15.23:47884) 2021-11-19T08:43:26.017 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.018+0000 7f996d279700 1 -- 172.21.15.23:0/2938224337 learned_addr learned my addr 172.21.15.23:0/2938224337 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:26.017 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.019+0000 7f996ca78700 1 --2- 172.21.15.23:0/2938224337 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99680fba30 0x7f9968109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:26.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.019+0000 7f996d279700 1 -- 172.21.15.23:0/2938224337 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99680fba30 msgr2=0x7f9968109430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:26.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.019+0000 7f996d279700 1 --2- 172.21.15.23:0/2938224337 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99680fba30 0x7f9968109430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:26.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.019+0000 7f996d279700 1 -- 172.21.15.23:0/2938224337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9964005040 con 0x7f9968060140 2021-11-19T08:43:26.019 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.019+0000 7f996d279700 1 --2- 172.21.15.23:0/2938224337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968060140 0x7f9968108f40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f9964009f50 tx=0x7f9964009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:26.019 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.019+0000 7f995dffb700 1 -- 172.21.15.23:0/2938224337 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9964018070 con 0x7f9968060140 2021-11-19T08:43:26.019 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.019+0000 7f995dffb700 1 -- 172.21.15.23:0/2938224337 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9964013410 con 0x7f9968060140 2021-11-19T08:43:26.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.019+0000 7f995dffb700 1 -- 172.21.15.23:0/2938224337 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f996401c620 con 0x7f9968060140 2021-11-19T08:43:26.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.019+0000 7f996e27b700 1 -- 172.21.15.23:0/2938224337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996810a460 con 0x7f9968060140 2021-11-19T08:43:26.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.019+0000 7f996e27b700 1 -- 172.21.15.23:0/2938224337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996810a9c0 con 0x7f9968060140 2021-11-19T08:43:26.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.020+0000 7f996e27b700 1 -- 172.21.15.23:0/2938224337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9968045ad0 con 0x7f9968060140 2021-11-19T08:43:26.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.021+0000 7f995dffb700 1 -- 172.21.15.23:0/2938224337 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f996401d420 con 0x7f9968060140 2021-11-19T08:43:26.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.021+0000 7f995dffb700 1 --2- 172.21.15.23:0/2938224337 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f995405c300 0x7f995405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:26.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.021+0000 7f995dffb700 1 -- 172.21.15.23:0/2938224337 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f9964077ff0 con 0x7f9968060140 2021-11-19T08:43:26.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.021+0000 7f996ca78700 1 --2- 172.21.15.23:0/2938224337 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f995405c300 0x7f995405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:26.023 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.021+0000 7f996ca78700 1 --2- 172.21.15.23:0/2938224337 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f995405c300 0x7f995405e7c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f996805ff10 tx=0x7f9958009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:26.024 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.025+0000 7f995dffb700 1 -- 172.21.15.23:0/2938224337 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f996404a950 con 0x7f9968060140 2021-11-19T08:43:26.209 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.224+0000 7f996e27b700 1 -- 172.21.15.23:0/2938224337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9968027a00 con 0x7f9968060140 2021-11-19T08:43:26.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.225+0000 7f995dffb700 1 -- 172.21.15.23:0/2938224337 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2580 (secure 0 0 0) 0x7f996404df70 con 0x7f9968060140 2021-11-19T08:43:26.211 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:26.211 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:23.229193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:26.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.227+0000 7f996e27b700 1 -- 172.21.15.23:0/2938224337 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f995405c300 msgr2=0x7f995405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:26.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.227+0000 7f996e27b700 1 --2- 172.21.15.23:0/2938224337 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f995405c300 0x7f995405e7c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f996805ff10 tx=0x7f9958009be0).stop 2021-11-19T08:43:26.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.227+0000 7f996e27b700 1 -- 172.21.15.23:0/2938224337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968060140 msgr2=0x7f9968108f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:26.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.227+0000 7f996e27b700 1 --2- 172.21.15.23:0/2938224337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968060140 0x7f9968108f40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f9964009f50 tx=0x7f9964009b30).stop 2021-11-19T08:43:26.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.227+0000 7f996e27b700 1 -- 172.21.15.23:0/2938224337 shutdown_connections 2021-11-19T08:43:26.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.227+0000 7f996e27b700 1 --2- 172.21.15.23:0/2938224337 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f995405c300 0x7f995405e7c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:26.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.227+0000 7f996e27b700 1 --2- 172.21.15.23:0/2938224337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968060140 0x7f9968108f40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:26.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.227+0000 7f996e27b700 1 --2- 172.21.15.23:0/2938224337 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99680fba30 0x7f9968109430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:26.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.227+0000 7f996e27b700 1 -- 172.21.15.23:0/2938224337 >> 172.21.15.23:0/2938224337 conn(0x7f99680f0ba0 msgr2=0x7f99680f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:26.217 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.227+0000 7f996e27b700 1 -- 172.21.15.23:0/2938224337 shutdown_connections 2021-11-19T08:43:26.218 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:26.227+0000 7f996e27b700 1 -- 172.21.15.23:0/2938224337 wait complete. 2021-11-19T08:43:26.219 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 63 2021-11-19T08:43:26.223 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:23.229193+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:26.224 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14582:up:clientreplay 2021-11-19T08:43:26.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:26 smithi134 conmon[36363]: cluster 2021-11-19T08:43:24.874429+0000 mgr.smithi023.ehlpcl (mgr.14164) 282 : cluster 2021-11-19T08:43:26.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:26 smithi134 conmon[36363]: [DBG] pgmap v253: 65 pgs: 65 active+clean; 4.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2021-11-19T08:43:26.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:26 smithi134 conmon[36363]: audit 2021-11-19T08:43:26.225867+0000 mon.smithi023 (mon.0) 733 : audit [DBG] from='client.? 172.21.15.23:0/2938224337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:26.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:26 smithi023 conmon[33490]: cluster 2021-11-19T08:43:24.874429+0000 mgr.smithi023.ehlpcl 2021-11-19T08:43:26.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:26 smithi023 conmon[33490]: (mgr.14164) 282 : cluster [DBG] pgmap v253: 65 pgs: 65 active+clean; 4.8 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2021-11-19T08:43:26.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:26 smithi023 conmon[33490]: audit 2021-11-19T08:43:26.225867+0000 mon.smithi023 (mon 2021-11-19T08:43:26.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:26 smithi023 conmon[33490]: .0) 733 : audit [DBG] from='client.? 172.21.15.23:0/2938224337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:27.225 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:27.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.378+0000 7fb2065fa700 1 -- 172.21.15.23:0/3473845868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2000f6680 msgr2=0x7fb2000f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:27.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.378+0000 7fb2065fa700 1 --2- 172.21.15.23:0/3473845868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2000f6680 0x7fb2000f6aa0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fb1f0004660 tx=0x7fb1f0009b30).stop 2021-11-19T08:43:27.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.378+0000 7fb2065fa700 1 -- 172.21.15.23:0/3473845868 shutdown_connections 2021-11-19T08:43:27.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.378+0000 7fb2065fa700 1 --2- 172.21.15.23:0/3473845868 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2000f7920 0x7fb2000f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:27.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.378+0000 7fb2065fa700 1 --2- 172.21.15.23:0/3473845868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2000f6680 0x7fb2000f6aa0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:27.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.378+0000 7fb2065fa700 1 -- 172.21.15.23:0/3473845868 >> 172.21.15.23:0/3473845868 conn(0x7fb2000f1de0 msgr2=0x7fb2000f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:27.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.378+0000 7fb2065fa700 1 -- 172.21.15.23:0/3473845868 shutdown_connections 2021-11-19T08:43:27.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.378+0000 7fb2065fa700 1 -- 172.21.15.23:0/3473845868 wait complete. 2021-11-19T08:43:27.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.379+0000 7fb2065fa700 1 Processor -- start 2021-11-19T08:43:27.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.379+0000 7fb2065fa700 1 -- start start 2021-11-19T08:43:27.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.379+0000 7fb2065fa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2000f6680 0x7fb20006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:27.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.379+0000 7fb2065fa700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2000f7920 0x7fb200069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:27.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.379+0000 7fb2065fa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20006bd70 con 0x7fb2000f6680 2021-11-19T08:43:27.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.379+0000 7fb2065fa700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20006a310 con 0x7fb2000f7920 2021-11-19T08:43:27.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.379+0000 7fb1fffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2000f6680 0x7fb20006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:27.369 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.379+0000 7fb1fffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2000f6680 0x7fb20006b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47896/0 (socket says 172.21.15.23:47896) 2021-11-19T08:43:27.369 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.379+0000 7fb1fffff700 1 -- 172.21.15.23:0/73801129 learned_addr learned my addr 172.21.15.23:0/73801129 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:27.369 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.379+0000 7fb1ff7fe700 1 --2- 172.21.15.23:0/73801129 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2000f7920 0x7fb200069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:27.369 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.380+0000 7fb1fffff700 1 -- 172.21.15.23:0/73801129 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2000f7920 msgr2=0x7fb200069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:27.370 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.380+0000 7fb1fffff700 1 --2- 172.21.15.23:0/73801129 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2000f7920 0x7fb200069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:27.370 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.380+0000 7fb1fffff700 1 -- 172.21.15.23:0/73801129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f0005040 con 0x7fb2000f6680 2021-11-19T08:43:27.370 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.380+0000 7fb1fffff700 1 --2- 172.21.15.23:0/73801129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2000f6680 0x7fb20006b7c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fb1f0009f50 tx=0x7fb1f0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:27.370 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.380+0000 7fb1fcff9700 1 -- 172.21.15.23:0/73801129 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1f0018070 con 0x7fb2000f6680 2021-11-19T08:43:27.370 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.380+0000 7fb1fcff9700 1 -- 172.21.15.23:0/73801129 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1f0013410 con 0x7fb2000f6680 2021-11-19T08:43:27.371 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.380+0000 7fb1fcff9700 1 -- 172.21.15.23:0/73801129 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1f001c3c0 con 0x7fb2000f6680 2021-11-19T08:43:27.371 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.380+0000 7fb2065fa700 1 -- 172.21.15.23:0/73801129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20006a530 con 0x7fb2000f6680 2021-11-19T08:43:27.371 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.380+0000 7fb2065fa700 1 -- 172.21.15.23:0/73801129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20006aa90 con 0x7fb2000f6680 2021-11-19T08:43:27.373 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.381+0000 7fb2065fa700 1 -- 172.21.15.23:0/73801129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb200046330 con 0x7fb2000f6680 2021-11-19T08:43:27.373 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.381+0000 7fb1fcff9700 1 -- 172.21.15.23:0/73801129 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb1f001d420 con 0x7fb2000f6680 2021-11-19T08:43:27.373 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.382+0000 7fb1fcff9700 1 --2- 172.21.15.23:0/73801129 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb1e805c2a0 0x7fb1e805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:27.373 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.382+0000 7fb1fcff9700 1 -- 172.21.15.23:0/73801129 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fb1f0077ea0 con 0x7fb2000f6680 2021-11-19T08:43:27.373 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.382+0000 7fb1ff7fe700 1 --2- 172.21.15.23:0/73801129 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb1e805c2a0 0x7fb1e805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:27.374 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.382+0000 7fb1ff7fe700 1 --2- 172.21.15.23:0/73801129 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb1e805c2a0 0x7fb1e805e760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fb1f4007fd0 tx=0x7fb1f4009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:27.374 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.386+0000 7fb1fcff9700 1 -- 172.21.15.23:0/73801129 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1f004a880 con 0x7fb2000f6680 2021-11-19T08:43:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.582+0000 7fb2065fa700 1 -- 172.21.15.23:0/73801129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2000fee10 con 0x7fb2000f6680 2021-11-19T08:43:27.567 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.582+0000 7fb1fcff9700 1 -- 172.21.15.23:0/73801129 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2580 (secure 0 0 0) 0x7fb1f004dea0 con 0x7fb2000f6680 2021-11-19T08:43:27.567 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:27.567 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:23.229193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:27.569 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.584+0000 7fb2065fa700 1 -- 172.21.15.23:0/73801129 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb1e805c2a0 msgr2=0x7fb1e805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:27.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.584+0000 7fb2065fa700 1 --2- 172.21.15.23:0/73801129 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb1e805c2a0 0x7fb1e805e760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fb1f4007fd0 tx=0x7fb1f4009040).stop 2021-11-19T08:43:27.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.584+0000 7fb2065fa700 1 -- 172.21.15.23:0/73801129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2000f6680 msgr2=0x7fb20006b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:27.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.584+0000 7fb2065fa700 1 --2- 172.21.15.23:0/73801129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2000f6680 0x7fb20006b7c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fb1f0009f50 tx=0x7fb1f0009b30).stop 2021-11-19T08:43:27.571 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.584+0000 7fb2065fa700 1 -- 172.21.15.23:0/73801129 shutdown_connections 2021-11-19T08:43:27.571 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.584+0000 7fb2065fa700 1 --2- 172.21.15.23:0/73801129 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb1e805c2a0 0x7fb1e805e760 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:27.571 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.584+0000 7fb2065fa700 1 --2- 172.21.15.23:0/73801129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2000f6680 0x7fb20006b7c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:27.571 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.584+0000 7fb2065fa700 1 --2- 172.21.15.23:0/73801129 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2000f7920 0x7fb200069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:27.571 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.584+0000 7fb2065fa700 1 -- 172.21.15.23:0/73801129 >> 172.21.15.23:0/73801129 conn(0x7fb2000f1de0 msgr2=0x7fb2000fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.584+0000 7fb2065fa700 1 -- 172.21.15.23:0/73801129 shutdown_connections 2021-11-19T08:43:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:27.584+0000 7fb2065fa700 1 -- 172.21.15.23:0/73801129 wait complete. 2021-11-19T08:43:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 63 2021-11-19T08:43:27.581 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:23.229193+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:27.582 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14582:up:clientreplay 2021-11-19T08:43:28.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:28 smithi134 conmon[36363]: cluster 2021-11-19T08:43:26.874883+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:43:28.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:28 smithi134 conmon[36363]: 283 : cluster [DBG] pgmap v254: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-19T08:43:28.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:28 smithi134 conmon[36363]: audit 2021-11-19T08:43:27.583351+0000 mon.smithi023 (mon.0) 734 : audit [DBG] 2021-11-19T08:43:28.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:28 smithi134 conmon[36363]: from='client.? 172.21.15.23:0/73801129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:28.583 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:28.611 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:28 smithi023 conmon[33490]: cluster 2021-11-19T08:43 2021-11-19T08:43:28.612 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:28 smithi023 conmon[33490]: :26.874883+0000 mgr.smithi023.ehlpcl (mgr.14164) 283 : cluster [DBG] pgmap v254: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-19T08:43:28.612 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:28 smithi023 conmon[33490]: audit 2021-11-19T08:43:27.583351+0000 mon.smithi023 (mon.0) 734 : audit [DBG] 2021-11-19T08:43:28.612 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:28 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/73801129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:28.718 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.731+0000 7f14f9930700 1 -- 172.21.15.23:0/2461823120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f40f6840 msgr2=0x7f14f40f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:28.718 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.731+0000 7f14f9930700 1 --2- 172.21.15.23:0/2461823120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f40f6840 0x7f14f40f6ca0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f14dc004660 tx=0x7f14dc009b30).stop 2021-11-19T08:43:28.719 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.731+0000 7f14f9930700 1 -- 172.21.15.23:0/2461823120 shutdown_connections 2021-11-19T08:43:28.719 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.731+0000 7f14f9930700 1 --2- 172.21.15.23:0/2461823120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f40f6840 0x7f14f40f6ca0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:28.719 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.731+0000 7f14f9930700 1 --2- 172.21.15.23:0/2461823120 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14f40f55a0 0x7f14f40f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:28.719 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.731+0000 7f14f9930700 1 -- 172.21.15.23:0/2461823120 >> 172.21.15.23:0/2461823120 conn(0x7f14f40f0ce0 msgr2=0x7f14f40f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:28.720 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.731+0000 7f14f9930700 1 -- 172.21.15.23:0/2461823120 shutdown_connections 2021-11-19T08:43:28.720 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.731+0000 7f14f9930700 1 -- 172.21.15.23:0/2461823120 wait complete. 2021-11-19T08:43:28.720 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.731+0000 7f14f9930700 1 Processor -- start 2021-11-19T08:43:28.720 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14f9930700 1 -- start start 2021-11-19T08:43:28.721 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14f9930700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f40f55a0 0x7f14f405ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:28.721 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14f9930700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14f40f6840 0x7f14f410bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:28.721 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14f9930700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f410c1a0 con 0x7f14f40f55a0 2021-11-19T08:43:28.721 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14f9930700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f410c2e0 con 0x7f14f40f6840 2021-11-19T08:43:28.721 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14f2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f40f55a0 0x7f14f405ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:28.722 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14f2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f40f55a0 0x7f14f405ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47906/0 (socket says 172.21.15.23:47906) 2021-11-19T08:43:28.722 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14f2ffd700 1 -- 172.21.15.23:0/469716364 learned_addr learned my addr 172.21.15.23:0/469716364 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:28.722 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14f27fc700 1 --2- 172.21.15.23:0/469716364 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14f40f6840 0x7f14f410bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:28.722 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14f2ffd700 1 -- 172.21.15.23:0/469716364 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14f40f6840 msgr2=0x7f14f410bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:28.723 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14f2ffd700 1 --2- 172.21.15.23:0/469716364 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14f40f6840 0x7f14f410bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:28.723 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14f2ffd700 1 -- 172.21.15.23:0/469716364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14dc005040 con 0x7f14f40f55a0 2021-11-19T08:43:28.723 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14f2ffd700 1 --2- 172.21.15.23:0/469716364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f40f55a0 0x7f14f405ff00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f14ec00d7f0 tx=0x7f14ec004e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:28.723 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14eb7fe700 1 -- 172.21.15.23:0/469716364 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14ec00f490 con 0x7f14f40f55a0 2021-11-19T08:43:28.724 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14eb7fe700 1 -- 172.21.15.23:0/469716364 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14ec00f040 con 0x7f14f40f55a0 2021-11-19T08:43:28.724 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.732+0000 7f14f9930700 1 -- 172.21.15.23:0/469716364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f410c540 con 0x7f14f40f55a0 2021-11-19T08:43:28.724 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.733+0000 7f14eb7fe700 1 -- 172.21.15.23:0/469716364 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14ec0139b0 con 0x7f14f40f55a0 2021-11-19T08:43:28.724 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.733+0000 7f14f9930700 1 -- 172.21.15.23:0/469716364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f410cad0 con 0x7f14f40f55a0 2021-11-19T08:43:28.724 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.734+0000 7f14f9930700 1 -- 172.21.15.23:0/469716364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f4046330 con 0x7f14f40f55a0 2021-11-19T08:43:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.734+0000 7f14eb7fe700 1 -- 172.21.15.23:0/469716364 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f14ec01a030 con 0x7f14f40f55a0 2021-11-19T08:43:28.726 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.734+0000 7f14eb7fe700 1 --2- 172.21.15.23:0/469716364 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f14e005c2a0 0x7f14e005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:28.726 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.734+0000 7f14eb7fe700 1 -- 172.21.15.23:0/469716364 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f14ec076040 con 0x7f14f40f55a0 2021-11-19T08:43:28.726 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.735+0000 7f14f27fc700 1 --2- 172.21.15.23:0/469716364 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f14e005c2a0 0x7f14e005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:28.728 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.735+0000 7f14f27fc700 1 --2- 172.21.15.23:0/469716364 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f14e005c2a0 0x7f14e005e760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f14dc009f50 tx=0x7f14dc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:28.728 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.738+0000 7f14eb7fe700 1 -- 172.21.15.23:0/469716364 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14ec0489a0 con 0x7f14f40f55a0 2021-11-19T08:43:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.934+0000 7f14f9930700 1 -- 172.21.15.23:0/469716364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14f4027a00 con 0x7f14f40f55a0 2021-11-19T08:43:28.919 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.935+0000 7f14eb7fe700 1 -- 172.21.15.23:0/469716364 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2580 (secure 0 0 0) 0x7f14ec018090 con 0x7f14f40f55a0 2021-11-19T08:43:28.920 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:28.920 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:23.229193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:28.921 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.936+0000 7f14f9930700 1 -- 172.21.15.23:0/469716364 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f14e005c2a0 msgr2=0x7f14e005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:28.921 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.936+0000 7f14f9930700 1 --2- 172.21.15.23:0/469716364 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f14e005c2a0 0x7f14e005e760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f14dc009f50 tx=0x7f14dc009b30).stop 2021-11-19T08:43:28.922 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.936+0000 7f14f9930700 1 -- 172.21.15.23:0/469716364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f40f55a0 msgr2=0x7f14f405ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:28.922 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.936+0000 7f14f9930700 1 --2- 172.21.15.23:0/469716364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f40f55a0 0x7f14f405ff00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f14ec00d7f0 tx=0x7f14ec004e80).stop 2021-11-19T08:43:28.922 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.936+0000 7f14f9930700 1 -- 172.21.15.23:0/469716364 shutdown_connections 2021-11-19T08:43:28.922 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.936+0000 7f14f9930700 1 --2- 172.21.15.23:0/469716364 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f14e005c2a0 0x7f14e005e760 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:28.923 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.936+0000 7f14f9930700 1 --2- 172.21.15.23:0/469716364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14f40f55a0 0x7f14f405ff00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:28.923 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.936+0000 7f14f9930700 1 --2- 172.21.15.23:0/469716364 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14f40f6840 0x7f14f410bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:28.923 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.936+0000 7f14f9930700 1 -- 172.21.15.23:0/469716364 >> 172.21.15.23:0/469716364 conn(0x7f14f40f0ce0 msgr2=0x7f14f40fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:28.923 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.936+0000 7f14f9930700 1 -- 172.21.15.23:0/469716364 shutdown_connections 2021-11-19T08:43:28.924 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:28.936+0000 7f14f9930700 1 -- 172.21.15.23:0/469716364 wait complete. 2021-11-19T08:43:28.924 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 63 2021-11-19T08:43:28.933 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:23.229193+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:28.933 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14582:up:clientreplay 2021-11-19T08:43:29.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:29 smithi134 conmon[36363]: audit 2021-11-19T08:43:28.935669+0000 mon.smithi023 (mon.0) 735 : audit 2021-11-19T08:43:29.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:29 smithi134 conmon[36363]: [DBG] from='client.? 172.21.15.23:0/469716364' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:29.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:29 smithi023 conmon[33490]: audit 2021-11-19T08:43:28.935669+0000 mon.smithi023 (mon.0) 735 : audit [DBG] from='client.? 172.21.15.23:0/469716364' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:29.934 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:30.070 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.084+0000 7fba7893f700 1 -- 172.21.15.23:0/1783304897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba740f96f0 msgr2=0x7fba740fbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:30.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.084+0000 7fba7893f700 1 --2- 172.21.15.23:0/1783304897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba740f96f0 0x7fba740fbb20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fba68004660 tx=0x7fba68009b30).stop 2021-11-19T08:43:30.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.084+0000 7fba7893f700 1 -- 172.21.15.23:0/1783304897 shutdown_connections 2021-11-19T08:43:30.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.084+0000 7fba7893f700 1 --2- 172.21.15.23:0/1783304897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba740f96f0 0x7fba740fbb20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:30.072 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.084+0000 7fba7893f700 1 --2- 172.21.15.23:0/1783304897 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba74060080 0x7fba740f9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:30.072 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.084+0000 7fba7893f700 1 -- 172.21.15.23:0/1783304897 >> 172.21.15.23:0/1783304897 conn(0x7fba740f0ba0 msgr2=0x7fba740f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:30.072 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.084+0000 7fba7893f700 1 -- 172.21.15.23:0/1783304897 shutdown_connections 2021-11-19T08:43:30.072 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.084+0000 7fba7893f700 1 -- 172.21.15.23:0/1783304897 wait complete. 2021-11-19T08:43:30.072 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.084+0000 7fba7893f700 1 Processor -- start 2021-11-19T08:43:30.073 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba7893f700 1 -- start start 2021-11-19T08:43:30.073 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba7893f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba74060080 0x7fba74106d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:30.073 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba7893f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba740f96f0 0x7fba74107250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:30.073 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba7893f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba74107850 con 0x7fba74060080 2021-11-19T08:43:30.074 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba7893f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba74108060 con 0x7fba740f96f0 2021-11-19T08:43:30.074 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba72d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba74060080 0x7fba74106d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:30.074 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba72d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba74060080 0x7fba74106d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47916/0 (socket says 172.21.15.23:47916) 2021-11-19T08:43:30.075 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba72d9d700 1 -- 172.21.15.23:0/4204788019 learned_addr learned my addr 172.21.15.23:0/4204788019 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:30.075 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba7259c700 1 --2- 172.21.15.23:0/4204788019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba740f96f0 0x7fba74107250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:30.075 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba72d9d700 1 -- 172.21.15.23:0/4204788019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba740f96f0 msgr2=0x7fba74107250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:30.075 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba72d9d700 1 --2- 172.21.15.23:0/4204788019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba740f96f0 0x7fba74107250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:30.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba72d9d700 1 -- 172.21.15.23:0/4204788019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba68005040 con 0x7fba74060080 2021-11-19T08:43:30.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba7259c700 1 --2- 172.21.15.23:0/4204788019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba740f96f0 0x7fba74107250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:43:30.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba72d9d700 1 --2- 172.21.15.23:0/4204788019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba74060080 0x7fba74106d60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fba6400e7f0 tx=0x7fba6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:30.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba5b7fe700 1 -- 172.21.15.23:0/4204788019 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba640092e0 con 0x7fba74060080 2021-11-19T08:43:30.077 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba7893f700 1 -- 172.21.15.23:0/4204788019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba741082e0 con 0x7fba74060080 2021-11-19T08:43:30.077 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba5b7fe700 1 -- 172.21.15.23:0/4204788019 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba6400f040 con 0x7fba74060080 2021-11-19T08:43:30.077 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.085+0000 7fba5b7fe700 1 -- 172.21.15.23:0/4204788019 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba640146c0 con 0x7fba74060080 2021-11-19T08:43:30.077 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.086+0000 7fba7893f700 1 -- 172.21.15.23:0/4204788019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba741088a0 con 0x7fba74060080 2021-11-19T08:43:30.078 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.087+0000 7fba7893f700 1 -- 172.21.15.23:0/4204788019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba740f4850 con 0x7fba74060080 2021-11-19T08:43:30.079 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.087+0000 7fba5b7fe700 1 -- 172.21.15.23:0/4204788019 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fba6401b030 con 0x7fba74060080 2021-11-19T08:43:30.079 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.087+0000 7fba5b7fe700 1 --2- 172.21.15.23:0/4204788019 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fba5c05c300 0x7fba5c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:30.079 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.087+0000 7fba5b7fe700 1 -- 172.21.15.23:0/4204788019 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fba64075fd0 con 0x7fba74060080 2021-11-19T08:43:30.079 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.087+0000 7fba7259c700 1 --2- 172.21.15.23:0/4204788019 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fba5c05c300 0x7fba5c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.088+0000 7fba7259c700 1 --2- 172.21.15.23:0/4204788019 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fba5c05c300 0x7fba5c05e7c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fba68000c00 tx=0x7fba6800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:30.080 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.091+0000 7fba5b7fe700 1 -- 172.21.15.23:0/4204788019 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba64048980 con 0x7fba74060080 2021-11-19T08:43:30.272 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.287+0000 7fba7893f700 1 -- 172.21.15.23:0/4204788019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba74045ad0 con 0x7fba74060080 2021-11-19T08:43:30.273 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.288+0000 7fba5b7fe700 1 -- 172.21.15.23:0/4204788019 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2580 (secure 0 0 0) 0x7fba64019020 con 0x7fba74060080 2021-11-19T08:43:30.274 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:30.274 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:23.229193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:30.276 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.289+0000 7fba7893f700 1 -- 172.21.15.23:0/4204788019 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fba5c05c300 msgr2=0x7fba5c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:30.279 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.289+0000 7fba7893f700 1 --2- 172.21.15.23:0/4204788019 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fba5c05c300 0x7fba5c05e7c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fba68000c00 tx=0x7fba6800b040).stop 2021-11-19T08:43:30.279 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.289+0000 7fba7893f700 1 -- 172.21.15.23:0/4204788019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba74060080 msgr2=0x7fba74106d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:30.280 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.289+0000 7fba7893f700 1 --2- 172.21.15.23:0/4204788019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba74060080 0x7fba74106d60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fba6400e7f0 tx=0x7fba6400b040).stop 2021-11-19T08:43:30.280 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.289+0000 7fba7893f700 1 -- 172.21.15.23:0/4204788019 shutdown_connections 2021-11-19T08:43:30.280 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.289+0000 7fba7893f700 1 --2- 172.21.15.23:0/4204788019 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fba5c05c300 0x7fba5c05e7c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:30.280 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.289+0000 7fba7893f700 1 --2- 172.21.15.23:0/4204788019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba74060080 0x7fba74106d60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:30.281 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.289+0000 7fba7893f700 1 --2- 172.21.15.23:0/4204788019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba740f96f0 0x7fba74107250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:30.281 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.289+0000 7fba7893f700 1 -- 172.21.15.23:0/4204788019 >> 172.21.15.23:0/4204788019 conn(0x7fba740f0ba0 msgr2=0x7fba740f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:30.281 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.290+0000 7fba7893f700 1 -- 172.21.15.23:0/4204788019 shutdown_connections 2021-11-19T08:43:30.281 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:30.290+0000 7fba7893f700 1 -- 172.21.15.23:0/4204788019 wait complete. 2021-11-19T08:43:30.282 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 63 2021-11-19T08:43:30.287 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:23.229193+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:30.288 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14582:up:clientreplay 2021-11-19T08:43:30.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:30 smithi134 conmon[36363]: cluster 2021-11-19T08:43:28.875253+0000 mgr.smithi023.ehlpcl (mgr.14164) 284 : cluster [DBG] pgmap v255: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-19T08:43:30.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:30 smithi023 conmon[33490]: cluster 2021-11-19T08:43:28.875253+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:43:30.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:30 smithi023 conmon[33490]: mgr.14164) 284 : cluster [DBG] pgmap v255: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-19T08:43:31.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:31.431 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.445+0000 7f0b3480a700 1 -- 172.21.15.23:0/2037098352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b300f6a70 msgr2=0x7f0b300f6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:31.431 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.445+0000 7f0b3480a700 1 --2- 172.21.15.23:0/2037098352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b300f6a70 0x7f0b300f6e90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f0b20004660 tx=0x7f0b20009b30).stop 2021-11-19T08:43:31.431 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.445+0000 7f0b3480a700 1 -- 172.21.15.23:0/2037098352 shutdown_connections 2021-11-19T08:43:31.431 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.445+0000 7f0b3480a700 1 --2- 172.21.15.23:0/2037098352 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b300f5770 0x7f0b300f5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:31.432 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.445+0000 7f0b3480a700 1 --2- 172.21.15.23:0/2037098352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b300f6a70 0x7f0b300f6e90 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:31.432 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.445+0000 7f0b3480a700 1 -- 172.21.15.23:0/2037098352 >> 172.21.15.23:0/2037098352 conn(0x7f0b300f0360 msgr2=0x7f0b300f27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:31.432 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.446+0000 7f0b3480a700 1 -- 172.21.15.23:0/2037098352 shutdown_connections 2021-11-19T08:43:31.432 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.446+0000 7f0b3480a700 1 -- 172.21.15.23:0/2037098352 wait complete. 2021-11-19T08:43:31.433 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.446+0000 7f0b3480a700 1 Processor -- start 2021-11-19T08:43:31.433 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.446+0000 7f0b3480a700 1 -- start start 2021-11-19T08:43:31.433 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.446+0000 7f0b3480a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b300f5770 0x7f0b30109a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:31.433 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.446+0000 7f0b3480a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b300f6a70 0x7f0b3010bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:31.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.446+0000 7f0b3480a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300f5400 con 0x7f0b300f6a70 2021-11-19T08:43:31.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.446+0000 7f0b3480a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300f5540 con 0x7f0b300f5770 2021-11-19T08:43:31.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.447+0000 7f0b2e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b300f6a70 0x7f0b3010bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:31.438 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.447+0000 7f0b2e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b300f6a70 0x7f0b3010bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47930/0 (socket says 172.21.15.23:47930) 2021-11-19T08:43:31.439 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.447+0000 7f0b2e59c700 1 -- 172.21.15.23:0/2827578741 learned_addr learned my addr 172.21.15.23:0/2827578741 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:31.440 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.447+0000 7f0b2ed9d700 1 --2- 172.21.15.23:0/2827578741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b300f5770 0x7f0b30109a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:31.442 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.447+0000 7f0b2e59c700 1 -- 172.21.15.23:0/2827578741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b300f5770 msgr2=0x7f0b30109a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:31.442 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.447+0000 7f0b2e59c700 1 --2- 172.21.15.23:0/2827578741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b300f5770 0x7f0b30109a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:31.442 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.447+0000 7f0b2e59c700 1 -- 172.21.15.23:0/2827578741 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b20005040 con 0x7f0b300f6a70 2021-11-19T08:43:31.443 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.447+0000 7f0b2e59c700 1 --2- 172.21.15.23:0/2827578741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b300f6a70 0x7f0b3010bf90 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f0b24002700 tx=0x7f0b24009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:31.443 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.447+0000 7f0b177fe700 1 -- 172.21.15.23:0/2827578741 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b240079c0 con 0x7f0b300f6a70 2021-11-19T08:43:31.443 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.447+0000 7f0b3480a700 1 -- 172.21.15.23:0/2827578741 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3010c540 con 0x7f0b300f6a70 2021-11-19T08:43:31.445 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.447+0000 7f0b177fe700 1 -- 172.21.15.23:0/2827578741 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b2400f040 con 0x7f0b300f6a70 2021-11-19T08:43:31.446 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.447+0000 7f0b177fe700 1 -- 172.21.15.23:0/2827578741 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b24014860 con 0x7f0b300f6a70 2021-11-19T08:43:31.446 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.447+0000 7f0b3480a700 1 -- 172.21.15.23:0/2827578741 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3010cad0 con 0x7f0b300f6a70 2021-11-19T08:43:31.446 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.448+0000 7f0b3480a700 1 -- 172.21.15.23:0/2827578741 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b300f4010 con 0x7f0b300f6a70 2021-11-19T08:43:31.448 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.448+0000 7f0b177fe700 1 -- 172.21.15.23:0/2827578741 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0b2401b060 con 0x7f0b300f6a70 2021-11-19T08:43:31.448 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.449+0000 7f0b177fe700 1 --2- 172.21.15.23:0/2827578741 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0b1805c300 0x7f0b1805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:31.449 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.449+0000 7f0b177fe700 1 -- 172.21.15.23:0/2827578741 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f0b240760e0 con 0x7f0b300f6a70 2021-11-19T08:43:31.449 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.449+0000 7f0b2ed9d700 1 --2- 172.21.15.23:0/2827578741 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0b1805c300 0x7f0b1805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:31.449 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.449+0000 7f0b2ed9d700 1 --2- 172.21.15.23:0/2827578741 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0b1805c300 0x7f0b1805e7c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f0b300f5080 tx=0x7f0b20009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:31.450 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.453+0000 7f0b177fe700 1 -- 172.21.15.23:0/2827578741 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b24048a40 con 0x7f0b300f6a70 2021-11-19T08:43:31.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:31 smithi134 conmon[36363]: audit 2021-11-19T08:43:30.288836+0000 mon.smithi023 (mon.0) 736 : audit [DBG] from='client.? 172.21.15.23:0/4204788019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:31.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.650+0000 7f0b3480a700 1 -- 172.21.15.23:0/2827578741 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b30045ad0 con 0x7f0b300f6a70 2021-11-19T08:43:31.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.650+0000 7f0b177fe700 1 -- 172.21.15.23:0/2827578741 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2580 (secure 0 0 0) 0x7f0b2404c060 con 0x7f0b300f6a70 2021-11-19T08:43:31.637 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:31.638 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:23.229193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:31.642 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.652+0000 7f0b3480a700 1 -- 172.21.15.23:0/2827578741 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0b1805c300 msgr2=0x7f0b1805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:31.643 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.652+0000 7f0b3480a700 1 --2- 172.21.15.23:0/2827578741 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0b1805c300 0x7f0b1805e7c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f0b300f5080 tx=0x7f0b20009b30).stop 2021-11-19T08:43:31.643 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.652+0000 7f0b3480a700 1 -- 172.21.15.23:0/2827578741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b300f6a70 msgr2=0x7f0b3010bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:31.644 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.652+0000 7f0b3480a700 1 --2- 172.21.15.23:0/2827578741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b300f6a70 0x7f0b3010bf90 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f0b24002700 tx=0x7f0b24009d80).stop 2021-11-19T08:43:31.644 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.652+0000 7f0b3480a700 1 -- 172.21.15.23:0/2827578741 shutdown_connections 2021-11-19T08:43:31.646 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.652+0000 7f0b3480a700 1 --2- 172.21.15.23:0/2827578741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b300f5770 0x7f0b30109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:31.646 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.652+0000 7f0b3480a700 1 --2- 172.21.15.23:0/2827578741 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0b1805c300 0x7f0b1805e7c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:31.647 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.652+0000 7f0b3480a700 1 --2- 172.21.15.23:0/2827578741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b300f6a70 0x7f0b3010bf90 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:31.647 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.652+0000 7f0b3480a700 1 -- 172.21.15.23:0/2827578741 >> 172.21.15.23:0/2827578741 conn(0x7f0b300f0360 msgr2=0x7f0b300fa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:31.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.652+0000 7f0b3480a700 1 -- 172.21.15.23:0/2827578741 shutdown_connections 2021-11-19T08:43:31.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:31.652+0000 7f0b3480a700 1 -- 172.21.15.23:0/2827578741 wait complete. 2021-11-19T08:43:31.648 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 63 2021-11-19T08:43:31.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:31 smithi023 conmon[33490]: audit 2021-11-19T08:43:30.288836+0000 mon.smithi023 (mon.0) 736 : audit [DBG] 2021-11-19T08:43:31.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:31 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/4204788019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:31.653 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:23.229193+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:31.653 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14582:up:clientreplay 2021-11-19T08:43:32.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:32 smithi134 conmon[36363]: cluster 2021-11-19T08:43:30.875791+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:43:32.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:32 smithi134 conmon[36363]: 14164) 285 : cluster [DBG] pgmap v256: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 351 B/s wr, 1 op/s 2021-11-19T08:43:32.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:32 smithi134 conmon[36363]: audit 2021-11-19T08:43:31.651231+0000 mon.smithi023 (mon.0) 737 : audit 2021-11-19T08:43:32.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:32 smithi134 conmon[36363]: [DBG] from='client.? 172.21.15.23:0/2827578741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:32.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:32 smithi023 conmon[33490]: cluster 2021-11-19T08:43:30.875791+0000 mgr.smithi023.ehlpcl (mgr.14164) 285 : cluster [DBG] 2021-11-19T08:43:32.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:32 smithi023 conmon[33490]: pgmap v256: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 351 B/s wr, 1 op/s 2021-11-19T08:43:32.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:32 smithi023 conmon[33490]: audit 2021-11-19T08:43:32.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:32 smithi023 conmon[33490]: 2021-11-19T08:43:31.651231+0000 mon.smithi023 (mon.0) 737 : audit [DBG] from='client.? 172.21.15.23:0/2827578741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:32.655 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:32.801 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.815+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/3106226508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a080f6840 msgr2=0x7f2a080f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:32.802 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.815+0000 7f2a0cf2b700 1 --2- 172.21.15.23:0/3106226508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a080f6840 0x7f2a080f6ca0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f29fc004660 tx=0x7f29fc009b30).stop 2021-11-19T08:43:32.802 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.815+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/3106226508 shutdown_connections 2021-11-19T08:43:32.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.815+0000 7f2a0cf2b700 1 --2- 172.21.15.23:0/3106226508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a080f6840 0x7f2a080f6ca0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:32.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.815+0000 7f2a0cf2b700 1 --2- 172.21.15.23:0/3106226508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a080f55a0 0x7f2a080f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:32.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.815+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/3106226508 >> 172.21.15.23:0/3106226508 conn(0x7f2a080f0ce0 msgr2=0x7f2a080f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:32.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.815+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/3106226508 shutdown_connections 2021-11-19T08:43:32.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.815+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/3106226508 wait complete. 2021-11-19T08:43:32.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.815+0000 7f2a0cf2b700 1 Processor -- start 2021-11-19T08:43:32.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.815+0000 7f2a0cf2b700 1 -- start start 2021-11-19T08:43:32.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.816+0000 7f2a0cf2b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a080f55a0 0x7f2a08109030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:32.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.816+0000 7f2a0cf2b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a080f6840 0x7f2a08109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:32.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.816+0000 7f2a0cf2b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a08109ad0 con 0x7f2a080f55a0 2021-11-19T08:43:32.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.816+0000 7f2a0cf2b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a08109c10 con 0x7f2a080f6840 2021-11-19T08:43:32.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.816+0000 7f2a0659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a080f55a0 0x7f2a08109030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:32.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.816+0000 7f2a0659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a080f55a0 0x7f2a08109030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47938/0 (socket says 172.21.15.23:47938) 2021-11-19T08:43:32.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.816+0000 7f2a0659c700 1 -- 172.21.15.23:0/671175422 learned_addr learned my addr 172.21.15.23:0/671175422 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:32.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.816+0000 7f2a05d9b700 1 --2- 172.21.15.23:0/671175422 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a080f6840 0x7f2a08109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:32.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.816+0000 7f2a0659c700 1 -- 172.21.15.23:0/671175422 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a080f6840 msgr2=0x7f2a08109520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:32.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.816+0000 7f2a0659c700 1 --2- 172.21.15.23:0/671175422 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a080f6840 0x7f2a08109520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:32.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.816+0000 7f2a0659c700 1 -- 172.21.15.23:0/671175422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc005040 con 0x7f2a080f55a0 2021-11-19T08:43:32.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.816+0000 7f2a0659c700 1 --2- 172.21.15.23:0/671175422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a080f55a0 0x7f2a08109030 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f29f800e7c0 tx=0x7f29f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:32.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.816+0000 7f29f6ffd700 1 -- 172.21.15.23:0/671175422 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29f80092e0 con 0x7f2a080f55a0 2021-11-19T08:43:32.808 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.817+0000 7f29f6ffd700 1 -- 172.21.15.23:0/671175422 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29f800f040 con 0x7f2a080f55a0 2021-11-19T08:43:32.808 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.817+0000 7f29f6ffd700 1 -- 172.21.15.23:0/671175422 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29f80147e0 con 0x7f2a080f55a0 2021-11-19T08:43:32.808 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.817+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/671175422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0810e6b0 con 0x7f2a080f55a0 2021-11-19T08:43:32.808 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.817+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/671175422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0810ec70 con 0x7f2a080f55a0 2021-11-19T08:43:32.809 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.817+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/671175422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a080fdd30 con 0x7f2a080f55a0 2021-11-19T08:43:32.810 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.818+0000 7f29f6ffd700 1 -- 172.21.15.23:0/671175422 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f29f801b030 con 0x7f2a080f55a0 2021-11-19T08:43:32.811 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.818+0000 7f29f6ffd700 1 --2- 172.21.15.23:0/671175422 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f29f005c2a0 0x7f29f005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:32.811 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.818+0000 7f29f6ffd700 1 -- 172.21.15.23:0/671175422 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6242+0+0 (secure 0 0 0) 0x7f29f80762f0 con 0x7f2a080f55a0 2021-11-19T08:43:32.811 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.818+0000 7f2a05d9b700 1 --2- 172.21.15.23:0/671175422 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f29f005c2a0 0x7f29f005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:32.811 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.819+0000 7f2a05d9b700 1 --2- 172.21.15.23:0/671175422 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f29f005c2a0 0x7f29f005e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f29fc004d80 tx=0x7f29fc00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:32.812 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:32.825+0000 7f29f6ffd700 1 -- 172.21.15.23:0/671175422 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29f8048c50 con 0x7f2a080f55a0 2021-11-19T08:43:33.016 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:33.032+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/671175422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a0810f310 con 0x7f2a080f55a0 2021-11-19T08:43:33.017 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:33.032+0000 7f29f6ffd700 1 -- 172.21.15.23:0/671175422 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+2580 (secure 0 0 0) 0x7f29f804c270 con 0x7f2a080f55a0 2021-11-19T08:43:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:33.017 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:23.229193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14582,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14582":{"gid":14582,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":56,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.23:6827/2135889102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2135889102},{"type":"v1","addr":"172.21.15.23:6827","nonce":2135889102}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:33.019 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:33.034+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/671175422 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f29f005c2a0 msgr2=0x7f29f005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:33.019 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:33.034+0000 7f2a0cf2b700 1 --2- 172.21.15.23:0/671175422 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f29f005c2a0 0x7f29f005e760 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f29fc004d80 tx=0x7f29fc00b040).stop 2021-11-19T08:43:33.020 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:33.034+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/671175422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a080f55a0 msgr2=0x7f2a08109030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:33.020 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:33.034+0000 7f2a0cf2b700 1 --2- 172.21.15.23:0/671175422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a080f55a0 0x7f2a08109030 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f29f800e7c0 tx=0x7f29f800b040).stop 2021-11-19T08:43:33.020 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:33.034+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/671175422 shutdown_connections 2021-11-19T08:43:33.020 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:33.034+0000 7f2a0cf2b700 1 --2- 172.21.15.23:0/671175422 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f29f005c2a0 0x7f29f005e760 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:33.020 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:33.034+0000 7f2a0cf2b700 1 --2- 172.21.15.23:0/671175422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a080f55a0 0x7f2a08109030 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:33.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:33.034+0000 7f2a0cf2b700 1 --2- 172.21.15.23:0/671175422 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a080f6840 0x7f2a08109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:33.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:33.034+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/671175422 >> 172.21.15.23:0/671175422 conn(0x7f2a080f0ce0 msgr2=0x7f2a0805fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:33.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:33.034+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/671175422 shutdown_connections 2021-11-19T08:43:33.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:33.034+0000 7f2a0cf2b700 1 -- 172.21.15.23:0/671175422 wait complete. 2021-11-19T08:43:33.022 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 63 2021-11-19T08:43:33.031 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:23.229193+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 51, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14582, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14582': {'gid': 14582, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 56, 'state': 'up:clientreplay', 'state_seq': 13, 'addr': '172.21.15.23:6827/2135889102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 2135889102}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 2135889102}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:33.032 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14582:up:clientreplay 2021-11-19T08:43:33.295 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:33 smithi023 conmon[33490]: audit 2021-11-19T08:43:33.033242+0000 2021-11-19T08:43:33.296 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:33 smithi023 conmon[33490]: mon.smithi023 (mon.0) 738 : audit [DBG] from='client.? 172.21.15.23:0/671175422' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:33.296 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:33 smithi023 conmon[33490]: cluster 2021-11-19T08:43:33.215949+0000 mon.smithi023 2021-11-19T08:43:33.296 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:33 smithi023 conmon[33490]: (mon.0) 739 : cluster [INF] daemon mds.cephfs.smithi023.fuvzkr restarted 2021-11-19T08:43:33.296 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:33 smithi023 conmon[33490]: cluster 2021-11-19T08:43:33. 2021-11-19T08:43:33.297 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:33 smithi023 conmon[33490]: 219503+0000 mon.smithi023 (mon.0) 740 : cluster [DBG] 2021-11-19T08:43:33.297 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:33 smithi023 conmon[33490]: osdmap e52: 6 total, 6 up, 6 in 2021-11-19T08:43:33.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:33 smithi134 conmon[36363]: audit 2021-11-19T08:43:33.033242+0000 mon.smithi023 (mon.0) 738 : audit 2021-11-19T08:43:33.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:33 smithi134 conmon[36363]: [DBG] from='client.? 172.21.15.23:0/671175422' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:33.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:33 smithi134 conmon[36363]: cluster 2021-11-19T08:43:33.215949+0000 mon.smithi023 (mon.0) 739 : cluster [INF] daemon mds.cephfs.smithi023.fuvzkr restarted 2021-11-19T08:43:33.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:33 smithi134 conmon[36363]: cluster 2021-11- 2021-11-19T08:43:33.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:33 smithi134 conmon[36363]: 19T08:43:33.219503+0000 mon.smithi023 (mon.0) 740 : cluster [DBG] osdmap e52: 6 total, 6 up, 6 in 2021-11-19T08:43:34.033 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:34.185 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.198+0000 7fc45da57700 1 -- 172.21.15.23:0/2970243633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4580f5580 msgr2=0x7fc4580f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:34.185 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.198+0000 7fc45da57700 1 --2- 172.21.15.23:0/2970243633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4580f5580 0x7fc4580f59a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fc448004660 tx=0x7fc448009b30).stop 2021-11-19T08:43:34.185 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc45da57700 1 -- 172.21.15.23:0/2970243633 shutdown_connections 2021-11-19T08:43:34.186 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc45da57700 1 --2- 172.21.15.23:0/2970243633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4580f6820 0x7fc4580f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:34.186 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc45da57700 1 --2- 172.21.15.23:0/2970243633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4580f5580 0x7fc4580f59a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:34.186 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc45da57700 1 -- 172.21.15.23:0/2970243633 >> 172.21.15.23:0/2970243633 conn(0x7fc4580f0ce0 msgr2=0x7fc4580f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:34.186 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc45da57700 1 -- 172.21.15.23:0/2970243633 shutdown_connections 2021-11-19T08:43:34.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc45da57700 1 -- 172.21.15.23:0/2970243633 wait complete. 2021-11-19T08:43:34.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc45da57700 1 Processor -- start 2021-11-19T08:43:34.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc45da57700 1 -- start start 2021-11-19T08:43:34.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc45da57700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4580f5580 0x7fc458105db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:34.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc45da57700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4580f6820 0x7fc4581062a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc45da57700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc458106850 con 0x7fc4580f6820 2021-11-19T08:43:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc45da57700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc458106990 con 0x7fc4580f5580 2021-11-19T08:43:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc44ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4580f6820 0x7fc4581062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc44ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4580f6820 0x7fc4581062a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47964/0 (socket says 172.21.15.23:47964) 2021-11-19T08:43:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.199+0000 7fc44ffff700 1 -- 172.21.15.23:0/3768098959 learned_addr learned my addr 172.21.15.23:0/3768098959 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.200+0000 7fc456ffd700 1 --2- 172.21.15.23:0/3768098959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4580f5580 0x7fc458105db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.200+0000 7fc44ffff700 1 -- 172.21.15.23:0/3768098959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4580f5580 msgr2=0x7fc458105db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.200+0000 7fc44ffff700 1 --2- 172.21.15.23:0/3768098959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4580f5580 0x7fc458105db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.200+0000 7fc44ffff700 1 -- 172.21.15.23:0/3768098959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc448005040 con 0x7fc4580f6820 2021-11-19T08:43:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.200+0000 7fc44ffff700 1 --2- 172.21.15.23:0/3768098959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4580f6820 0x7fc4581062a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fc4400026d0 tx=0x7fc440009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.200+0000 7fc44f7fe700 1 -- 172.21.15.23:0/3768098959 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc440009320 con 0x7fc4580f6820 2021-11-19T08:43:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.200+0000 7fc44f7fe700 1 -- 172.21.15.23:0/3768098959 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc44000f040 con 0x7fc4580f6820 2021-11-19T08:43:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.200+0000 7fc44f7fe700 1 -- 172.21.15.23:0/3768098959 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4400147e0 con 0x7fc4580f6820 2021-11-19T08:43:34.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.200+0000 7fc45da57700 1 -- 172.21.15.23:0/3768098959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45810b430 con 0x7fc4580f6820 2021-11-19T08:43:34.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.200+0000 7fc45da57700 1 -- 172.21.15.23:0/3768098959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45810b9f0 con 0x7fc4580f6820 2021-11-19T08:43:34.192 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.201+0000 7fc45da57700 1 -- 172.21.15.23:0/3768098959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc458027a00 con 0x7fc4580f6820 2021-11-19T08:43:34.192 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.201+0000 7fc44f7fe700 1 -- 172.21.15.23:0/3768098959 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc440009480 con 0x7fc4580f6820 2021-11-19T08:43:34.193 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.202+0000 7fc44f7fe700 1 --2- 172.21.15.23:0/3768098959 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc44405c2a0 0x7fc44405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:34.193 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.202+0000 7fc44f7fe700 1 -- 172.21.15.23:0/3768098959 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc4400768f0 con 0x7fc4580f6820 2021-11-19T08:43:34.193 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.202+0000 7fc456ffd700 1 --2- 172.21.15.23:0/3768098959 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc44405c2a0 0x7fc44405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:34.193 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.202+0000 7fc456ffd700 1 --2- 172.21.15.23:0/3768098959 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc44405c2a0 0x7fc44405e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fc448004e30 tx=0x7fc448009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:34.193 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.206+0000 7fc44f7fe700 1 -- 172.21.15.23:0/3768098959 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc440049280 con 0x7fc4580f6820 2021-11-19T08:43:34.388 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.403+0000 7fc45da57700 1 -- 172.21.15.23:0/3768098959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4580fdd10 con 0x7fc4580f6820 2021-11-19T08:43:34.388 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.404+0000 7fc44f7fe700 1 -- 172.21.15.23:0/3768098959 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2576 (secure 0 0 0) 0x7fc44004c8a0 con 0x7fc4580f6820 2021-11-19T08:43:34.388 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:34.389 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":64}],"filesystems":[{"mdsmap":{"epoch":66,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:34.283331+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:resolve","state_seq":10,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:34.390 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.405+0000 7fc45da57700 1 -- 172.21.15.23:0/3768098959 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc44405c2a0 msgr2=0x7fc44405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:34.390 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.405+0000 7fc45da57700 1 --2- 172.21.15.23:0/3768098959 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc44405c2a0 0x7fc44405e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fc448004e30 tx=0x7fc448009b30).stop 2021-11-19T08:43:34.390 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.405+0000 7fc45da57700 1 -- 172.21.15.23:0/3768098959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4580f6820 msgr2=0x7fc4581062a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:34.391 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.405+0000 7fc45da57700 1 --2- 172.21.15.23:0/3768098959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4580f6820 0x7fc4581062a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fc4400026d0 tx=0x7fc440009de0).stop 2021-11-19T08:43:34.391 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.405+0000 7fc45da57700 1 -- 172.21.15.23:0/3768098959 shutdown_connections 2021-11-19T08:43:34.391 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.405+0000 7fc45da57700 1 --2- 172.21.15.23:0/3768098959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc4580f5580 0x7fc458105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:34.391 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.405+0000 7fc45da57700 1 --2- 172.21.15.23:0/3768098959 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc44405c2a0 0x7fc44405e760 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:34.392 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.405+0000 7fc45da57700 1 --2- 172.21.15.23:0/3768098959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4580f6820 0x7fc4581062a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:34.392 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.405+0000 7fc45da57700 1 -- 172.21.15.23:0/3768098959 >> 172.21.15.23:0/3768098959 conn(0x7fc4580f0ce0 msgr2=0x7fc4580fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:34.392 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.405+0000 7fc45da57700 1 -- 172.21.15.23:0/3768098959 shutdown_connections 2021-11-19T08:43:34.392 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:34.405+0000 7fc45da57700 1 -- 172.21.15.23:0/3768098959 wait complete. 2021-11-19T08:43:34.393 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 66 2021-11-19T08:43:34.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:34 smithi023 conmon[33490]: cluster 2021-11-19T08:43:32.876190+0000 mgr.smithi023.ehlpcl (mgr.14164) 286 : cluster 2021-11-19T08:43:34.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:34 smithi023 conmon[33490]: [DBG] pgmap v257: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 341 B/s wr, 0 op/s 2021-11-19T08:43:34.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:34 smithi023 conmon[33490]: cluster 2021-11-19T08:43:33.281510+0000 mon.smithi023 (mon.0) 741 : cluster [DBG] mds.? [v2:172.21.15.23:6826/4052910066,v1:172.21.15.23:6827/4052910066] up:boot 2021-11-19T08:43:34.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:34 smithi023 conmon[33490]: cluster 2021-11-19T08:43:33.281580 2021-11-19T08:43:34.394 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:34 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 742 : cluster [INF] Standby daemon mds.cephfs.smithi134.jqsnzo assigned to filesystem cephfs as rank 0 2021-11-19T08:43:34.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:34 smithi023 conmon[33490]: cluster 2021-11-19T08:43:33.281828+0000 mon.smithi023 ( 2021-11-19T08:43:34.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:34 smithi023 conmon[33490]: mon.0) 743 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi134.opxllb=up:active} 2 up:standby, 1 failed 2021-11-19T08:43:34.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:34 smithi023 conmon[33490]: audit 2021-11-19T08:43:33 2021-11-19T08:43:34.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:34 smithi023 conmon[33490]: .282039+0000 mon.smithi023 (mon.0) 744 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.fuvzkr"}]: dispatch 2021-11-19T08:43:34.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:34 smithi023 conmon[33490]: cluster 2021- 2021-11-19T08:43:34.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:34 smithi023 conmon[33490]: 11-19T08:43:33.284050+0000 mon.smithi023 (mon.0) 745 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:replay,1=cephfs.smithi134.opxllb=up:active} 1 up:standby 2021-11-19T08:43:34.402 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:34.283331+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:resolve', 'state_seq': 10, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:34.402 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:resolve 2021-11-19T08:43:34.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:34 smithi134 conmon[36363]: cluster 2021-11-19T08:43:32.876190+0000 mgr.smithi023.ehlpcl (mgr.14164) 286 : cluster [DBG] pgmap v257: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 341 B/s wr, 0 op/s 2021-11-19T08:43:34.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:34 smithi134 conmon[36363]: cluster 2021-11-19T08 2021-11-19T08:43:34.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:34 smithi134 conmon[36363]: :43:33.281510+0000 mon.smithi023 (mon.0) 741 : cluster [DBG] mds.? [v2:172.21.15.23:6826/4052910066,v1:172.21.15.23:6827/4052910066] up:boot 2021-11-19T08:43:34.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:34 smithi134 conmon[36363]: cluster 2021-11-19T08:43:33.281580+0000 mon.smithi023 (mon.0) 742 : cluster [INF] Standby daemon mds.cephfs.smithi134.jqsnzo assigned to filesystem cephfs as rank 0 2021-11-19T08:43:34.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:34 smithi134 conmon[36363]: cluster 2021-11-19T08:43 2021-11-19T08:43:34.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:34 smithi134 conmon[36363]: :33.281828+0000 mon.smithi023 (mon.0) 743 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi134.opxllb=up:active} 2 up:standby, 1 failed 2021-11-19T08:43:34.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:34 smithi134 conmon[36363]: audit 2021-11-19T08:43:33. 2021-11-19T08:43:34.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:34 smithi134 conmon[36363]: 282039+0000 mon.smithi023 (mon.0) 744 : audit [DBG] 2021-11-19T08:43:34.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:34 smithi134 conmon[36363]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.fuvzkr"}]: dispatch 2021-11-19T08:43:34.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:34 smithi134 conmon[36363]: cluster 2021-11-19T08:43: 2021-11-19T08:43:34.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:34 smithi134 conmon[36363]: 33.284050+0000 mon.smithi023 (mon.0) 745 : cluster 2021-11-19T08:43:34.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:34 smithi134 conmon[36363]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:replay,1=cephfs.smithi134.opxllb=up:active} 1 up:standby 2021-11-19T08:43:35.403 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:35.540 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.552+0000 7fa9d8287700 1 -- 172.21.15.23:0/1435748731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9d00f6b70 msgr2=0x7fa9d00f6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:35.540 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.552+0000 7fa9d8287700 1 --2- 172.21.15.23:0/1435748731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9d00f6b70 0x7fa9d00f6f90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fa9cc004660 tx=0x7fa9cc009b30).stop 2021-11-19T08:43:35.541 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d8287700 1 -- 172.21.15.23:0/1435748731 shutdown_connections 2021-11-19T08:43:35.541 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d8287700 1 --2- 172.21.15.23:0/1435748731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d00f5870 0x7fa9d00f5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:35.541 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d8287700 1 --2- 172.21.15.23:0/1435748731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9d00f6b70 0x7fa9d00f6f90 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:35.541 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d8287700 1 -- 172.21.15.23:0/1435748731 >> 172.21.15.23:0/1435748731 conn(0x7fa9d00f0440 msgr2=0x7fa9d00f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:35.542 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d8287700 1 -- 172.21.15.23:0/1435748731 shutdown_connections 2021-11-19T08:43:35.542 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d8287700 1 -- 172.21.15.23:0/1435748731 wait complete. 2021-11-19T08:43:35.542 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d8287700 1 Processor -- start 2021-11-19T08:43:35.542 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d8287700 1 -- start start 2021-11-19T08:43:35.543 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d8287700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d00f5870 0x7fa9d00f5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:35.543 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d8287700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9d00f6b70 0x7fa9d010bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:35.543 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d8287700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d010c1c0 con 0x7fa9d00f6b70 2021-11-19T08:43:35.543 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d8287700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d010c300 con 0x7fa9d00f5870 2021-11-19T08:43:35.544 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d5822700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9d00f6b70 0x7fa9d010bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:35.544 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d5822700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9d00f6b70 0x7fa9d010bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47976/0 (socket says 172.21.15.23:47976) 2021-11-19T08:43:35.544 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.553+0000 7fa9d5822700 1 -- 172.21.15.23:0/2601781933 learned_addr learned my addr 172.21.15.23:0/2601781933 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:35.545 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.554+0000 7fa9d6023700 1 --2- 172.21.15.23:0/2601781933 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d00f5870 0x7fa9d00f5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:35.545 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.554+0000 7fa9d5822700 1 -- 172.21.15.23:0/2601781933 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d00f5870 msgr2=0x7fa9d00f5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:35.545 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.554+0000 7fa9d5822700 1 --2- 172.21.15.23:0/2601781933 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d00f5870 0x7fa9d00f5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:35.545 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.554+0000 7fa9d5822700 1 -- 172.21.15.23:0/2601781933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9cc005040 con 0x7fa9d00f6b70 2021-11-19T08:43:35.545 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.554+0000 7fa9d5822700 1 --2- 172.21.15.23:0/2601781933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9d00f6b70 0x7fa9d010bcd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fa9c0002700 tx=0x7fa9c0009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:35.546 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.554+0000 7fa9c6ffd700 1 -- 172.21.15.23:0/2601781933 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9c0009320 con 0x7fa9d00f6b70 2021-11-19T08:43:35.546 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.554+0000 7fa9c6ffd700 1 -- 172.21.15.23:0/2601781933 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9c000f040 con 0x7fa9d00f6b70 2021-11-19T08:43:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.554+0000 7fa9d8287700 1 -- 172.21.15.23:0/2601781933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d010c560 con 0x7fa9d00f6b70 2021-11-19T08:43:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.554+0000 7fa9c6ffd700 1 -- 172.21.15.23:0/2601781933 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9c0014900 con 0x7fa9d00f6b70 2021-11-19T08:43:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.554+0000 7fa9d8287700 1 -- 172.21.15.23:0/2601781933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d010caf0 con 0x7fa9d00f6b70 2021-11-19T08:43:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.555+0000 7fa9d8287700 1 -- 172.21.15.23:0/2601781933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d0046330 con 0x7fa9d00f6b70 2021-11-19T08:43:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.555+0000 7fa9c6ffd700 1 -- 172.21.15.23:0/2601781933 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa9c0009480 con 0x7fa9d00f6b70 2021-11-19T08:43:35.548 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.556+0000 7fa9c6ffd700 1 --2- 172.21.15.23:0/2601781933 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa9bc05c2a0 0x7fa9bc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:35.548 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.556+0000 7fa9c6ffd700 1 -- 172.21.15.23:0/2601781933 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa9c00765c0 con 0x7fa9d00f6b70 2021-11-19T08:43:35.548 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.556+0000 7fa9d6023700 1 --2- 172.21.15.23:0/2601781933 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa9bc05c2a0 0x7fa9bc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:35.549 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:35 smithi134 conmon[36363]: cluster 2021-11-19T08:43:34.285794+0000 mon.smithi023 (mon.0) 746 : cluster [DBG] mds.? [v2:172.21.15.134:6824/464801254,v1:172.21.15.134:6825/464801254] up:resolve 2021-11-19T08:43:35.549 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:35 smithi134 conmon[36363]: cluster 2021-11-19T08:43:34.285848+0000 mon.smithi023 (mon.0) 747 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi134.opxllb=up:active} 1 up:standby 2021-11-19T08:43:35.550 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:35 smithi134 conmon[36363]: audit 2021-11-19T08:43:34.404649+0000 mon.smithi023 (mon.0) 748 : audit [DBG] 2021-11-19T08:43:35.550 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:35 smithi134 conmon[36363]: from='client.? 172.21.15.23:0/3768098959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.556+0000 7fa9d6023700 1 --2- 172.21.15.23:0/2601781933 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa9bc05c2a0 0x7fa9bc05e760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fa9cc009d60 tx=0x7fa9cc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:35.551 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.560+0000 7fa9c6ffd700 1 -- 172.21.15.23:0/2601781933 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9c0048ed0 con 0x7fa9d00f6b70 2021-11-19T08:43:35.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: cluster 2021-11-19T08:43:35.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 2021 2021-11-19T08:43:35.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: - 2021-11-19T08:43:35.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 11- 2021-11-19T08:43:35.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 19 2021-11-19T08:43:35.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: T 2021-11-19T08:43:35.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 08 2021-11-19T08:43:35.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: : 2021-11-19T08:43:35.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 43 2021-11-19T08:43:35.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: :34 2021-11-19T08:43:35.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: . 2021-11-19T08:43:35.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 285794+0000 2021-11-19T08:43:35.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 2021-11-19T08:43:35.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: mon.smithi023 ( 2021-11-19T08:43:35.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: mon 2021-11-19T08:43:35.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: . 2021-11-19T08:43:35.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 0) 2021-11-19T08:43:35.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 746 2021-11-19T08:43:35.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: : cluster 2021-11-19T08:43:35.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: [DBG] 2021-11-19T08:43:35.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 2021-11-19T08:43:35.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: mds.? [v2:172.21.15.134:6824/464801254,v1:172.21.15.134:6825/464801254] up:resolve 2021-11-19T08:43:35.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 2021-11-19T08:43:35.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: cluster 2021-11-19T08:43:35.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 2021-11-19T08:43:35.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 2021 2021-11-19T08:43:35.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: - 2021-11-19T08:43:35.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 11 2021-11-19T08:43:35.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: - 2021-11-19T08:43:35.651 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 19 2021-11-19T08:43:35.652 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: T 2021-11-19T08:43:35.652 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 08 2021-11-19T08:43:35.652 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: : 2021-11-19T08:43:35.652 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 43 2021-11-19T08:43:35.653 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: : 2021-11-19T08:43:35.653 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 34 2021-11-19T08:43:35.653 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: . 2021-11-19T08:43:35.653 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 285848 2021-11-19T08:43:35.653 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: +0000 2021-11-19T08:43:35.654 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: mon.smithi023 2021-11-19T08:43:35.654 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: ( 2021-11-19T08:43:35.654 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: mon 2021-11-19T08:43:35.654 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: . 2021-11-19T08:43:35.655 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 0 2021-11-19T08:43:35.655 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: ) 747 2021-11-19T08:43:35.655 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: : cluster 2021-11-19T08:43:35.655 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 2021-11-19T08:43:35.656 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: [DBG] 2021-11-19T08:43:35.656 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 2021-11-19T08:43:35.656 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi134.opxllb=up:active} 1 up:standby 2021-11-19T08:43:35.656 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 2021-11-19T08:43:35.657 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: audit 2021-11-19T08:43:35.657 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 2021-11-19T08:43:35.657 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 2021 2021-11-19T08:43:35.657 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: - 2021-11-19T08:43:35.657 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 11- 2021-11-19T08:43:35.658 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 19 2021-11-19T08:43:35.658 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: T 2021-11-19T08:43:35.658 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 08 2021-11-19T08:43:35.658 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: : 2021-11-19T08:43:35.658 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 43 2021-11-19T08:43:35.659 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: : 2021-11-19T08:43:35.659 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 34.404649 2021-11-19T08:43:35.659 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: +0000 2021-11-19T08:43:35.659 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 2021-11-19T08:43:35.660 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: mon.smithi023 2021-11-19T08:43:35.660 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: (mon 2021-11-19T08:43:35.660 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: . 2021-11-19T08:43:35.660 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 0 2021-11-19T08:43:35.660 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: ) 2021-11-19T08:43:35.661 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 748 2021-11-19T08:43:35.661 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: : 2021-11-19T08:43:35.661 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: audit 2021-11-19T08:43:35.661 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: [DBG] 2021-11-19T08:43:35.662 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 2021-11-19T08:43:35.662 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/3768098959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:35.662 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:35 smithi023 conmon[33490]: 2021-11-19T08:43:35.741 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.756+0000 7fa9d8287700 1 -- 172.21.15.23:0/2601781933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9d00fdd60 con 0x7fa9d00f6b70 2021-11-19T08:43:35.741 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.757+0000 7fa9c6ffd700 1 -- 172.21.15.23:0/2601781933 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2576 (secure 0 0 0) 0x7fa9c004c4f0 con 0x7fa9d00f6b70 2021-11-19T08:43:35.742 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:35.742 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":64}],"filesystems":[{"mdsmap":{"epoch":66,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:34.283331+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:resolve","state_seq":10,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:35.744 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.759+0000 7fa9d8287700 1 -- 172.21.15.23:0/2601781933 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa9bc05c2a0 msgr2=0x7fa9bc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:35.744 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.759+0000 7fa9d8287700 1 --2- 172.21.15.23:0/2601781933 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa9bc05c2a0 0x7fa9bc05e760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fa9cc009d60 tx=0x7fa9cc009b30).stop 2021-11-19T08:43:35.744 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.759+0000 7fa9d8287700 1 -- 172.21.15.23:0/2601781933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9d00f6b70 msgr2=0x7fa9d010bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:35.744 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.759+0000 7fa9d8287700 1 --2- 172.21.15.23:0/2601781933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9d00f6b70 0x7fa9d010bcd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fa9c0002700 tx=0x7fa9c0009d80).stop 2021-11-19T08:43:35.745 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.759+0000 7fa9d8287700 1 -- 172.21.15.23:0/2601781933 shutdown_connections 2021-11-19T08:43:35.745 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.759+0000 7fa9d8287700 1 --2- 172.21.15.23:0/2601781933 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa9d00f5870 0x7fa9d00f5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:35.745 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.759+0000 7fa9d8287700 1 --2- 172.21.15.23:0/2601781933 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa9bc05c2a0 0x7fa9bc05e760 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:35.745 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.759+0000 7fa9d8287700 1 --2- 172.21.15.23:0/2601781933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9d00f6b70 0x7fa9d010bcd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:35.745 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.759+0000 7fa9d8287700 1 -- 172.21.15.23:0/2601781933 >> 172.21.15.23:0/2601781933 conn(0x7fa9d00f0440 msgr2=0x7fa9d00fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:35.746 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.759+0000 7fa9d8287700 1 -- 172.21.15.23:0/2601781933 shutdown_connections 2021-11-19T08:43:35.746 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:35.759+0000 7fa9d8287700 1 -- 172.21.15.23:0/2601781933 wait complete. 2021-11-19T08:43:35.746 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 66 2021-11-19T08:43:35.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:34.283331+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:resolve', 'state_seq': 10, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:35.764 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:resolve 2021-11-19T08:43:36.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:36 smithi023 conmon[33490]: cluster 2021-11-19T08:43:34.876651+0000 mgr.smithi023.ehlpcl (mgr.14164) 287 : cluster [DBG] pgmap v259: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 307 B/s wr, 1 op/s 2021-11-19T08:43:36.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:36 smithi023 conmon[33490]: audit 2021-11-19T08:43:35.757988+0000 mon.smithi023 (mon.0) 749 : audit [DBG] from='client.? 172.21.15.23:0/2601781933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:36.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:36 smithi134 conmon[36363]: cluster 2021-11-19T08:43:34.876651+0000 mgr.smithi023.ehlpcl (mgr.14164) 287 : cluster [DBG] pgmap v259: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 307 B/s wr, 1 op/s 2021-11-19T08:43:36.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:36 smithi134 conmon[36363]: audit 2021-11-19T08:43:35.757988+0000 mon.smithi023 (mon.0) 749 : audit [DBG] from='client.? 172.21.15.23:0/2601781933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:36.765 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:36.900 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.914+0000 7f8be1f30700 1 -- 172.21.15.23:0/3037047942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bdc0f7950 msgr2=0x7f8bdc0f7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:36.900 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.914+0000 7f8be1f30700 1 --2- 172.21.15.23:0/3037047942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bdc0f7950 0x7f8bdc0f7db0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f8bd0004660 tx=0x7f8bd0009b30).stop 2021-11-19T08:43:36.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.914+0000 7f8be1f30700 1 -- 172.21.15.23:0/3037047942 shutdown_connections 2021-11-19T08:43:36.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.914+0000 7f8be1f30700 1 --2- 172.21.15.23:0/3037047942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bdc0f7950 0x7f8bdc0f7db0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:36.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.914+0000 7f8be1f30700 1 --2- 172.21.15.23:0/3037047942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bdc0f66b0 0x7f8bdc0f6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:36.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.914+0000 7f8be1f30700 1 -- 172.21.15.23:0/3037047942 >> 172.21.15.23:0/3037047942 conn(0x7f8bdc0f1e10 msgr2=0x7f8bdc0f4230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:36.902 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.914+0000 7f8be1f30700 1 -- 172.21.15.23:0/3037047942 shutdown_connections 2021-11-19T08:43:36.902 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.914+0000 7f8be1f30700 1 -- 172.21.15.23:0/3037047942 wait complete. 2021-11-19T08:43:36.902 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.914+0000 7f8be1f30700 1 Processor -- start 2021-11-19T08:43:36.902 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.915+0000 7f8be1f30700 1 -- start start 2021-11-19T08:43:36.903 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.915+0000 7f8be1f30700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bdc0f66b0 0x7f8bdc106e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:36.903 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.915+0000 7f8be1f30700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bdc0f7950 0x7f8bdc107350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:36.903 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.915+0000 7f8be1f30700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc107900 con 0x7f8bdc0f66b0 2021-11-19T08:43:36.903 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.915+0000 7f8be1f30700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc107a40 con 0x7f8bdc0f7950 2021-11-19T08:43:36.903 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.915+0000 7f8bdb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bdc0f66b0 0x7f8bdc106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:36.904 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.915+0000 7f8bdb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bdc0f66b0 0x7f8bdc106e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47984/0 (socket says 172.21.15.23:47984) 2021-11-19T08:43:36.904 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.915+0000 7f8bdb7fe700 1 -- 172.21.15.23:0/864561365 learned_addr learned my addr 172.21.15.23:0/864561365 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:36.904 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.915+0000 7f8bdaffd700 1 --2- 172.21.15.23:0/864561365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bdc0f7950 0x7f8bdc107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:36.904 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.915+0000 7f8bdb7fe700 1 -- 172.21.15.23:0/864561365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bdc0f7950 msgr2=0x7f8bdc107350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:36.904 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.915+0000 7f8bdb7fe700 1 --2- 172.21.15.23:0/864561365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bdc0f7950 0x7f8bdc107350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:36.905 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.915+0000 7f8bdb7fe700 1 -- 172.21.15.23:0/864561365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd0005040 con 0x7f8bdc0f66b0 2021-11-19T08:43:36.905 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.915+0000 7f8bdb7fe700 1 --2- 172.21.15.23:0/864561365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bdc0f66b0 0x7f8bdc106e60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f8bcc00c790 tx=0x7f8bcc007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:36.905 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.916+0000 7f8bc3fff700 1 -- 172.21.15.23:0/864561365 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bcc013630 con 0x7f8bdc0f66b0 2021-11-19T08:43:36.906 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.916+0000 7f8be1f30700 1 -- 172.21.15.23:0/864561365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bdc10c4e0 con 0x7f8bdc0f66b0 2021-11-19T08:43:36.906 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.916+0000 7f8bc3fff700 1 -- 172.21.15.23:0/864561365 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8bcc012860 con 0x7f8bdc0f66b0 2021-11-19T08:43:36.906 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.916+0000 7f8bc3fff700 1 -- 172.21.15.23:0/864561365 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bcc002820 con 0x7f8bdc0f66b0 2021-11-19T08:43:36.907 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.916+0000 7f8be1f30700 1 -- 172.21.15.23:0/864561365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bdc10caa0 con 0x7f8bdc0f66b0 2021-11-19T08:43:36.908 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.917+0000 7f8be1f30700 1 -- 172.21.15.23:0/864561365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bdc027a00 con 0x7f8bdc0f66b0 2021-11-19T08:43:36.909 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.917+0000 7f8bc3fff700 1 -- 172.21.15.23:0/864561365 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8bcc010070 con 0x7f8bdc0f66b0 2021-11-19T08:43:36.909 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.917+0000 7f8bc3fff700 1 --2- 172.21.15.23:0/864561365 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8bc405c2a0 0x7f8bc405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:36.909 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.917+0000 7f8bc3fff700 1 -- 172.21.15.23:0/864561365 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8bcc0762f0 con 0x7f8bdc0f66b0 2021-11-19T08:43:36.909 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.917+0000 7f8bdaffd700 1 --2- 172.21.15.23:0/864561365 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8bc405c2a0 0x7f8bc405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:36.909 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.918+0000 7f8bdaffd700 1 --2- 172.21.15.23:0/864561365 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8bc405c2a0 0x7f8bc405e760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f8bd0004630 tx=0x7f8bd0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:36.910 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:36.921+0000 7f8bc3fff700 1 -- 172.21.15.23:0/864561365 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8bcc048c00 con 0x7f8bdc0f66b0 2021-11-19T08:43:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:37.116+0000 7f8be1f30700 1 -- 172.21.15.23:0/864561365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bdc0fee40 con 0x7f8bdc0f66b0 2021-11-19T08:43:37.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:37.116+0000 7f8bc3fff700 1 -- 172.21.15.23:0/864561365 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2576 (secure 0 0 0) 0x7f8bcc04c220 con 0x7f8bdc0f66b0 2021-11-19T08:43:37.101 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:37.102 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":64}],"filesystems":[{"mdsmap":{"epoch":66,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:34.283331+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:resolve","state_seq":10,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:37.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:37.118+0000 7f8be1f30700 1 -- 172.21.15.23:0/864561365 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8bc405c2a0 msgr2=0x7f8bc405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:37.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:37.118+0000 7f8be1f30700 1 --2- 172.21.15.23:0/864561365 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8bc405c2a0 0x7f8bc405e760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f8bd0004630 tx=0x7f8bd0009b30).stop 2021-11-19T08:43:37.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:37.118+0000 7f8be1f30700 1 -- 172.21.15.23:0/864561365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bdc0f66b0 msgr2=0x7f8bdc106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:37.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:37.118+0000 7f8be1f30700 1 --2- 172.21.15.23:0/864561365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bdc0f66b0 0x7f8bdc106e60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f8bcc00c790 tx=0x7f8bcc007d90).stop 2021-11-19T08:43:37.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:37.118+0000 7f8be1f30700 1 -- 172.21.15.23:0/864561365 shutdown_connections 2021-11-19T08:43:37.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:37.118+0000 7f8be1f30700 1 --2- 172.21.15.23:0/864561365 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8bc405c2a0 0x7f8bc405e760 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:37.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:37.118+0000 7f8be1f30700 1 --2- 172.21.15.23:0/864561365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bdc0f66b0 0x7f8bdc106e60 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:37.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:37.118+0000 7f8be1f30700 1 --2- 172.21.15.23:0/864561365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bdc0f7950 0x7f8bdc107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:37.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:37.118+0000 7f8be1f30700 1 -- 172.21.15.23:0/864561365 >> 172.21.15.23:0/864561365 conn(0x7f8bdc0f1e10 msgr2=0x7f8bdc0fb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:37.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:37.118+0000 7f8be1f30700 1 -- 172.21.15.23:0/864561365 shutdown_connections 2021-11-19T08:43:37.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:37.118+0000 7f8be1f30700 1 -- 172.21.15.23:0/864561365 wait complete. 2021-11-19T08:43:37.106 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 66 2021-11-19T08:43:37.114 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:34.283331+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:resolve', 'state_seq': 10, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:37.115 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:resolve 2021-11-19T08:43:37.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:37 smithi023 conmon[33490]: audit 2021-11-19T08:43:37.117477+0000 mon.smithi023 (mon. 2021-11-19T08:43:37.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:37 smithi023 conmon[33490]: 0) 750 : audit [DBG] from='client.? 172.21.15.23:0/864561365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:37.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:37 smithi134 conmon[36363]: audit 2021-11-19T08:43:37.117477+0000 mon.smithi023 (mon.0) 750 : audit [DBG] from='client.? 172.21.15.23:0/864561365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:38.116 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.265+0000 7f68bfd3a700 1 -- 172.21.15.23:0/363512070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80f55b0 msgr2=0x7f68b80f59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.265+0000 7f68bfd3a700 1 --2- 172.21.15.23:0/363512070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80f55b0 0x7f68b80f59d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f68b0004660 tx=0x7f68b0009b30).stop 2021-11-19T08:43:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.265+0000 7f68bfd3a700 1 -- 172.21.15.23:0/363512070 shutdown_connections 2021-11-19T08:43:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.265+0000 7f68bfd3a700 1 --2- 172.21.15.23:0/363512070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68b80f6850 0x7f68b80f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.265+0000 7f68bfd3a700 1 --2- 172.21.15.23:0/363512070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80f55b0 0x7f68b80f59d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.265+0000 7f68bfd3a700 1 -- 172.21.15.23:0/363512070 >> 172.21.15.23:0/363512070 conn(0x7f68b80f0d10 msgr2=0x7f68b80f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.265+0000 7f68bfd3a700 1 -- 172.21.15.23:0/363512070 shutdown_connections 2021-11-19T08:43:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.265+0000 7f68bfd3a700 1 -- 172.21.15.23:0/363512070 wait complete. 2021-11-19T08:43:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.265+0000 7f68bfd3a700 1 Processor -- start 2021-11-19T08:43:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bfd3a700 1 -- start start 2021-11-19T08:43:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bfd3a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80f55b0 0x7f68b8105de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bfd3a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68b80f6850 0x7f68b81062d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:38.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bfd3a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b8106880 con 0x7f68b80f55b0 2021-11-19T08:43:38.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bfd3a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b81069c0 con 0x7f68b80f6850 2021-11-19T08:43:38.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bdad6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80f55b0 0x7f68b8105de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:38.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bdad6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80f55b0 0x7f68b8105de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47996/0 (socket says 172.21.15.23:47996) 2021-11-19T08:43:38.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bdad6700 1 -- 172.21.15.23:0/945007999 learned_addr learned my addr 172.21.15.23:0/945007999 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:38.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bd2d5700 1 --2- 172.21.15.23:0/945007999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68b80f6850 0x7f68b81062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:38.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bdad6700 1 -- 172.21.15.23:0/945007999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68b80f6850 msgr2=0x7f68b81062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:38.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bdad6700 1 --2- 172.21.15.23:0/945007999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68b80f6850 0x7f68b81062d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:38.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bdad6700 1 -- 172.21.15.23:0/945007999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b0005040 con 0x7f68b80f55b0 2021-11-19T08:43:38.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bdad6700 1 --2- 172.21.15.23:0/945007999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80f55b0 0x7f68b8105de0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f68b0009f50 tx=0x7f68b0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:38.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68b67fc700 1 -- 172.21.15.23:0/945007999 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68b001a070 con 0x7f68b80f55b0 2021-11-19T08:43:38.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bfd3a700 1 -- 172.21.15.23:0/945007999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b805fa00 con 0x7f68b80f55b0 2021-11-19T08:43:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68b67fc700 1 -- 172.21.15.23:0/945007999 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68b0012a80 con 0x7f68b80f55b0 2021-11-19T08:43:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68b67fc700 1 -- 172.21.15.23:0/945007999 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68b001f900 con 0x7f68b80f55b0 2021-11-19T08:43:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.266+0000 7f68bfd3a700 1 -- 172.21.15.23:0/945007999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b805ffc0 con 0x7f68b80f55b0 2021-11-19T08:43:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.267+0000 7f68bfd3a700 1 -- 172.21.15.23:0/945007999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b8046330 con 0x7f68b80f55b0 2021-11-19T08:43:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.268+0000 7f68b67fc700 1 -- 172.21.15.23:0/945007999 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f68b0012bf0 con 0x7f68b80f55b0 2021-11-19T08:43:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.268+0000 7f68b67fc700 1 --2- 172.21.15.23:0/945007999 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f68a005c2a0 0x7f68a005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.268+0000 7f68b67fc700 1 -- 172.21.15.23:0/945007999 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f68b007a400 con 0x7f68b80f55b0 2021-11-19T08:43:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.268+0000 7f68bd2d5700 1 --2- 172.21.15.23:0/945007999 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f68a005c2a0 0x7f68a005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.269+0000 7f68bd2d5700 1 --2- 172.21.15.23:0/945007999 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f68a005c2a0 0x7f68a005e760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f68ac0096c0 tx=0x7f68ac009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.272+0000 7f68b67fc700 1 -- 172.21.15.23:0/945007999 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68b004cd10 con 0x7f68b80f55b0 2021-11-19T08:43:38.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.469+0000 7f68bfd3a700 1 -- 172.21.15.23:0/945007999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68b80ffd80 con 0x7f68b80f55b0 2021-11-19T08:43:38.455 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.470+0000 7f68b67fc700 1 -- 172.21.15.23:0/945007999 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f68b0050330 con 0x7f68b80f55b0 2021-11-19T08:43:38.457 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:38.457 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:34.283331+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:resolve","state_seq":10,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:38.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.471+0000 7f68bfd3a700 1 -- 172.21.15.23:0/945007999 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f68a005c2a0 msgr2=0x7f68a005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:38.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.471+0000 7f68bfd3a700 1 --2- 172.21.15.23:0/945007999 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f68a005c2a0 0x7f68a005e760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f68ac0096c0 tx=0x7f68ac009be0).stop 2021-11-19T08:43:38.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.471+0000 7f68bfd3a700 1 -- 172.21.15.23:0/945007999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80f55b0 msgr2=0x7f68b8105de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:38.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.471+0000 7f68bfd3a700 1 --2- 172.21.15.23:0/945007999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80f55b0 0x7f68b8105de0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f68b0009f50 tx=0x7f68b0009b30).stop 2021-11-19T08:43:38.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.471+0000 7f68bfd3a700 1 -- 172.21.15.23:0/945007999 shutdown_connections 2021-11-19T08:43:38.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.471+0000 7f68bfd3a700 1 --2- 172.21.15.23:0/945007999 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f68a005c2a0 0x7f68a005e760 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:38.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.471+0000 7f68bfd3a700 1 --2- 172.21.15.23:0/945007999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80f55b0 0x7f68b8105de0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:38.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.471+0000 7f68bfd3a700 1 --2- 172.21.15.23:0/945007999 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68b80f6850 0x7f68b81062d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:38.464 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.471+0000 7f68bfd3a700 1 -- 172.21.15.23:0/945007999 >> 172.21.15.23:0/945007999 conn(0x7f68b80f0d10 msgr2=0x7f68b80fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:38.464 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.472+0000 7f68bfd3a700 1 -- 172.21.15.23:0/945007999 shutdown_connections 2021-11-19T08:43:38.464 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:38.472+0000 7f68bfd3a700 1 -- 172.21.15.23:0/945007999 wait complete. 2021-11-19T08:43:38.464 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 67 2021-11-19T08:43:38.466 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:38 smithi023 conmon[33490]: cluster 2021-11-19T08:43:36.877082+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:43:38.467 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:38 smithi023 conmon[33490]: 14164) 288 : cluster [DBG] pgmap v260: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:38.467 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:38 smithi023 conmon[33490]: cluster 2021-11-19T08:43:37.318455+0000 mon.smithi023 (mon.0) 751 : cluster [DBG] mds.? [v2:172.21.15.23:6826/4052910066,v1:172.21.15.23:6827/4052910066] up:standby 2021-11-19T08:43:38.468 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:38 smithi023 conmon[33490]: cluster 2021-11-19T08:43:37.318519+0000 mon.smithi023 (mon.0) 752 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi134.opxllb=up:active} 1 up:standby 2021-11-19T08:43:38.468 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:38 smithi023 conmon[33490]: 2021-11-19T08:43:38.470 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:34.283331+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:resolve', 'state_seq': 10, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:38.470 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:resolve 2021-11-19T08:43:38.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:38 smithi134 conmon[36363]: cluster 2021-11-19T08:43:36.877082+0000 mgr.smithi023.ehlpcl (mgr.14164) 288 : cluster [DBG] pgmap v260: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:38.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:38 smithi134 conmon[36363]: cluster 2021-11-19T08:43:37.318455+0000 mon.smithi023 (mon.0) 751 : cluster [DBG] mds.? [v2:172.21.15.23:6826/4052910066,v1:172.21.15.23:6827/4052910066] up:standby 2021-11-19T08:43:38.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:38 smithi134 conmon[36363]: cluster 2021-11-19T08 2021-11-19T08:43:38.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:38 smithi134 conmon[36363]: :43:37.318519+0000 mon.smithi023 (mon.0) 752 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi134.opxllb=up:active} 1 up:standby 2021-11-19T08:43:39.472 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:39.606 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.619+0000 7f69bb59e700 1 -- 172.21.15.23:0/3582587969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69bc0f52a0 msgr2=0x7f69bc0f7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:39.606 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.619+0000 7f69bb59e700 1 --2- 172.21.15.23:0/3582587969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69bc0f52a0 0x7f69bc0f7be0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f69b0004660 tx=0x7f69b0009b30).stop 2021-11-19T08:43:39.607 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69bb59e700 1 -- 172.21.15.23:0/3582587969 shutdown_connections 2021-11-19T08:43:39.607 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69bb59e700 1 --2- 172.21.15.23:0/3582587969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69bc0f52a0 0x7f69bc0f7be0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:39.607 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69bb59e700 1 --2- 172.21.15.23:0/3582587969 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69bc0f4900 0x7f69bc0f4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:39.608 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69bb59e700 1 -- 172.21.15.23:0/3582587969 >> 172.21.15.23:0/3582587969 conn(0x7f69bc0f0360 msgr2=0x7f69bc0f27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:39.608 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69bb59e700 1 -- 172.21.15.23:0/3582587969 shutdown_connections 2021-11-19T08:43:39.608 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69bb59e700 1 -- 172.21.15.23:0/3582587969 wait complete. 2021-11-19T08:43:39.608 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69bb59e700 1 Processor -- start 2021-11-19T08:43:39.608 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69bb59e700 1 -- start start 2021-11-19T08:43:39.609 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69bb59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69bc0f4900 0x7f69bc106f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:39.609 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69bb59e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69bc0f52a0 0x7f69bc107460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:39.609 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69bb59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc107a10 con 0x7f69bc0f4900 2021-11-19T08:43:39.609 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69bb59e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc107b50 con 0x7f69bc0f52a0 2021-11-19T08:43:39.610 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69ba59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69bc0f4900 0x7f69bc106f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:39.610 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69ba59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69bc0f4900 0x7f69bc106f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48006/0 (socket says 172.21.15.23:48006) 2021-11-19T08:43:39.610 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69ba59c700 1 -- 172.21.15.23:0/2355970099 learned_addr learned my addr 172.21.15.23:0/2355970099 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:39.610 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.620+0000 7f69b9d9b700 1 --2- 172.21.15.23:0/2355970099 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69bc0f52a0 0x7f69bc107460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:39.611 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.621+0000 7f69ba59c700 1 -- 172.21.15.23:0/2355970099 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69bc0f52a0 msgr2=0x7f69bc107460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:39.611 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.621+0000 7f69ba59c700 1 --2- 172.21.15.23:0/2355970099 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69bc0f52a0 0x7f69bc107460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:39.612 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.621+0000 7f69ba59c700 1 -- 172.21.15.23:0/2355970099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b0005040 con 0x7f69bc0f4900 2021-11-19T08:43:39.612 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.621+0000 7f69ba59c700 1 --2- 172.21.15.23:0/2355970099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69bc0f4900 0x7f69bc106f70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f69ac0026d0 tx=0x7f69ac009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:39.612 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.621+0000 7f69aaffd700 1 -- 172.21.15.23:0/2355970099 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69ac0079c0 con 0x7f69bc0f4900 2021-11-19T08:43:39.612 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.621+0000 7f69aaffd700 1 -- 172.21.15.23:0/2355970099 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69ac00f040 con 0x7f69bc0f4900 2021-11-19T08:43:39.613 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.621+0000 7f69bb59e700 1 -- 172.21.15.23:0/2355970099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69bc1084f0 con 0x7f69bc0f4900 2021-11-19T08:43:39.613 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.621+0000 7f69aaffd700 1 -- 172.21.15.23:0/2355970099 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69ac0148b0 con 0x7f69bc0f4900 2021-11-19T08:43:39.613 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.621+0000 7f69bb59e700 1 -- 172.21.15.23:0/2355970099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69bc108ab0 con 0x7f69bc0f4900 2021-11-19T08:43:39.613 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.622+0000 7f69bb59e700 1 -- 172.21.15.23:0/2355970099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69bc045ad0 con 0x7f69bc0f4900 2021-11-19T08:43:39.615 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.622+0000 7f69aaffd700 1 -- 172.21.15.23:0/2355970099 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f69ac007b20 con 0x7f69bc0f4900 2021-11-19T08:43:39.615 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.623+0000 7f69aaffd700 1 --2- 172.21.15.23:0/2355970099 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f69a405c300 0x7f69a405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:39.615 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.623+0000 7f69aaffd700 1 -- 172.21.15.23:0/2355970099 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f69ac0761f0 con 0x7f69bc0f4900 2021-11-19T08:43:39.616 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.623+0000 7f69b9d9b700 1 --2- 172.21.15.23:0/2355970099 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f69a405c300 0x7f69a405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:39.616 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.623+0000 7f69b9d9b700 1 --2- 172.21.15.23:0/2355970099 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f69a405c300 0x7f69a405e7c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f69b0009f50 tx=0x7f69b0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:39.616 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.627+0000 7f69aaffd700 1 -- 172.21.15.23:0/2355970099 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69ac048c30 con 0x7f69bc0f4900 2021-11-19T08:43:39.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:39 smithi023 conmon[33490]: audit 2021-11-19T08:43:38.471009+0000 mon.smithi023 (mon.0) 753 : 2021-11-19T08:43:39.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:39 smithi023 conmon[33490]: audit [DBG] from='client.? 172.21.15.23:0/945007999' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:39.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:39 smithi134 conmon[36363]: audit 2021-11-19T08:43:38.471009+0000 mon.smithi023 (mon.0) 753 : audit [DBG] from='client.? 172.21.15.23:0/945007999' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:39.808 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.822+0000 7f69bb59e700 1 -- 172.21.15.23:0/2355970099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69bc05f580 con 0x7f69bc0f4900 2021-11-19T08:43:39.809 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.823+0000 7f69aaffd700 1 -- 172.21.15.23:0/2355970099 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f69ac04c250 con 0x7f69bc0f4900 2021-11-19T08:43:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:39.810 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:34.283331+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:resolve","state_seq":10,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:39.812 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.824+0000 7f69bb59e700 1 -- 172.21.15.23:0/2355970099 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f69a405c300 msgr2=0x7f69a405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:39.813 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.824+0000 7f69bb59e700 1 --2- 172.21.15.23:0/2355970099 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f69a405c300 0x7f69a405e7c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f69b0009f50 tx=0x7f69b0009b30).stop 2021-11-19T08:43:39.813 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.824+0000 7f69bb59e700 1 -- 172.21.15.23:0/2355970099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69bc0f4900 msgr2=0x7f69bc106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:39.813 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.824+0000 7f69bb59e700 1 --2- 172.21.15.23:0/2355970099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69bc0f4900 0x7f69bc106f70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f69ac0026d0 tx=0x7f69ac009de0).stop 2021-11-19T08:43:39.814 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.825+0000 7f69bb59e700 1 -- 172.21.15.23:0/2355970099 shutdown_connections 2021-11-19T08:43:39.814 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.825+0000 7f69bb59e700 1 --2- 172.21.15.23:0/2355970099 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f69a405c300 0x7f69a405e7c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:39.814 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.825+0000 7f69bb59e700 1 --2- 172.21.15.23:0/2355970099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69bc0f4900 0x7f69bc106f70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:39.815 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.825+0000 7f69bb59e700 1 --2- 172.21.15.23:0/2355970099 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69bc0f52a0 0x7f69bc107460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:39.815 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.825+0000 7f69bb59e700 1 -- 172.21.15.23:0/2355970099 >> 172.21.15.23:0/2355970099 conn(0x7f69bc0f0360 msgr2=0x7f69bc0fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:39.816 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.825+0000 7f69bb59e700 1 -- 172.21.15.23:0/2355970099 shutdown_connections 2021-11-19T08:43:39.816 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:39.825+0000 7f69bb59e700 1 -- 172.21.15.23:0/2355970099 wait complete. 2021-11-19T08:43:39.816 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 67 2021-11-19T08:43:39.821 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:34.283331+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:resolve', 'state_seq': 10, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:39.821 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:resolve 2021-11-19T08:43:40.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:40 smithi023 conmon[33490]: cluster 2021-11-19T08:43:38.877496+0000 mgr.smithi023.ehlpcl (mgr.14164) 289 : cluster [DBG] pgmap v261: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:40.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:40 smithi023 conmon[33490]: audit 2021-11-19T08:43:39.823977+0000 mon.smithi023 (mon.0) 754 : audit [DBG] from='client.? 172.21.15.23:0/2355970099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:40.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:40 smithi134 conmon[36363]: cluster 2021-11-19T08:43:38.877496+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:43:40.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:40 smithi134 conmon[36363]: 14164) 289 : cluster [DBG] pgmap v261: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:40.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:40 smithi134 conmon[36363]: audit 2021-11-19T08:43:39.823977+0000 mon.smithi023 (mon.0) 754 : audit [DBG] from='client.? 172.21.15.23:0/2355970099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:40.823 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:40.957 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.971+0000 7f1a15593700 1 -- 172.21.15.23:0/1206507012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a100f6840 msgr2=0x7f1a100f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:40.957 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.971+0000 7f1a15593700 1 --2- 172.21.15.23:0/1206507012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a100f6840 0x7f1a100f6ca0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f1a04004660 tx=0x7f1a04009b30).stop 2021-11-19T08:43:40.958 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.971+0000 7f1a15593700 1 -- 172.21.15.23:0/1206507012 shutdown_connections 2021-11-19T08:43:40.958 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.971+0000 7f1a15593700 1 --2- 172.21.15.23:0/1206507012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a100f6840 0x7f1a100f6ca0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:40.958 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.971+0000 7f1a15593700 1 --2- 172.21.15.23:0/1206507012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a100f55a0 0x7f1a100f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:40.958 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.971+0000 7f1a15593700 1 -- 172.21.15.23:0/1206507012 >> 172.21.15.23:0/1206507012 conn(0x7f1a100f0ce0 msgr2=0x7f1a100f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:40.958 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.971+0000 7f1a15593700 1 -- 172.21.15.23:0/1206507012 shutdown_connections 2021-11-19T08:43:40.959 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.971+0000 7f1a15593700 1 -- 172.21.15.23:0/1206507012 wait complete. 2021-11-19T08:43:40.959 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.971+0000 7f1a15593700 1 Processor -- start 2021-11-19T08:43:40.959 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.971+0000 7f1a15593700 1 -- start start 2021-11-19T08:43:40.959 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f1a15593700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a100f55a0 0x7f1a10105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:40.960 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f1a15593700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a100f6840 0x7f1a10106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:40.960 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f1a15593700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a10106840 con 0x7f1a100f55a0 2021-11-19T08:43:40.961 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f1a15593700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a10106980 con 0x7f1a100f6840 2021-11-19T08:43:40.961 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f1a0effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a100f55a0 0x7f1a10105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:40.962 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f1a0effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a100f55a0 0x7f1a10105da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48016/0 (socket says 172.21.15.23:48016) 2021-11-19T08:43:40.962 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f1a0effd700 1 -- 172.21.15.23:0/2145754691 learned_addr learned my addr 172.21.15.23:0/2145754691 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:40.962 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f1a0e7fc700 1 --2- 172.21.15.23:0/2145754691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a100f6840 0x7f1a10106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:40.962 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f1a0effd700 1 -- 172.21.15.23:0/2145754691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a100f6840 msgr2=0x7f1a10106290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:40.963 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f1a0effd700 1 --2- 172.21.15.23:0/2145754691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a100f6840 0x7f1a10106290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:40.963 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f1a0effd700 1 -- 172.21.15.23:0/2145754691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a04005040 con 0x7f1a100f55a0 2021-11-19T08:43:40.963 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f1a0effd700 1 --2- 172.21.15.23:0/2145754691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a100f55a0 0x7f1a10105da0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f1a0800d7f0 tx=0x7f1a08004e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:40.963 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f19f77fe700 1 -- 172.21.15.23:0/2145754691 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a0800f490 con 0x7f1a100f55a0 2021-11-19T08:43:40.963 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f1a15593700 1 -- 172.21.15.23:0/2145754691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a1010b420 con 0x7f1a100f55a0 2021-11-19T08:43:40.964 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f19f77fe700 1 -- 172.21.15.23:0/2145754691 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a0800f040 con 0x7f1a100f55a0 2021-11-19T08:43:40.964 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.972+0000 7f19f77fe700 1 -- 172.21.15.23:0/2145754691 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a080139b0 con 0x7f1a100f55a0 2021-11-19T08:43:40.964 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.973+0000 7f1a15593700 1 -- 172.21.15.23:0/2145754691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1010b9e0 con 0x7f1a100f55a0 2021-11-19T08:43:40.966 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.973+0000 7f1a15593700 1 -- 172.21.15.23:0/2145754691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a10027a00 con 0x7f1a100f55a0 2021-11-19T08:43:40.966 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.974+0000 7f19f77fe700 1 -- 172.21.15.23:0/2145754691 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1a0801a030 con 0x7f1a100f55a0 2021-11-19T08:43:40.966 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.974+0000 7f19f77fe700 1 --2- 172.21.15.23:0/2145754691 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f19f805c2a0 0x7f19f805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:40.967 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.974+0000 7f19f77fe700 1 -- 172.21.15.23:0/2145754691 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1a080760a0 con 0x7f1a100f55a0 2021-11-19T08:43:40.967 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.974+0000 7f1a0e7fc700 1 --2- 172.21.15.23:0/2145754691 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f19f805c2a0 0x7f19f805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:40.967 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.975+0000 7f1a0e7fc700 1 --2- 172.21.15.23:0/2145754691 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f19f805c2a0 0x7f19f805e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f1a04009f50 tx=0x7f1a04009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:40.967 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:40.978+0000 7f19f77fe700 1 -- 172.21.15.23:0/2145754691 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a08048950 con 0x7f1a100f55a0 2021-11-19T08:43:41.158 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:41.174+0000 7f1a15593700 1 -- 172.21.15.23:0/2145754691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a100fdd30 con 0x7f1a100f55a0 2021-11-19T08:43:41.159 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:41.174+0000 7f19f77fe700 1 -- 172.21.15.23:0/2145754691 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f1a08018090 con 0x7f1a100f55a0 2021-11-19T08:43:41.159 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:41.159 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:34.283331+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:resolve","state_seq":10,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:41.161 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:41.175+0000 7f1a15593700 1 -- 172.21.15.23:0/2145754691 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f19f805c2a0 msgr2=0x7f19f805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:41.161 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:41.176+0000 7f1a15593700 1 --2- 172.21.15.23:0/2145754691 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f19f805c2a0 0x7f19f805e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f1a04009f50 tx=0x7f1a04009b30).stop 2021-11-19T08:43:41.161 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:41.176+0000 7f1a15593700 1 -- 172.21.15.23:0/2145754691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a100f55a0 msgr2=0x7f1a10105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:41.161 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:41.176+0000 7f1a15593700 1 --2- 172.21.15.23:0/2145754691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a100f55a0 0x7f1a10105da0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f1a0800d7f0 tx=0x7f1a08004e80).stop 2021-11-19T08:43:41.162 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:41.176+0000 7f1a15593700 1 -- 172.21.15.23:0/2145754691 shutdown_connections 2021-11-19T08:43:41.162 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:41.176+0000 7f1a15593700 1 --2- 172.21.15.23:0/2145754691 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f19f805c2a0 0x7f19f805e760 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:41.162 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:41.176+0000 7f1a15593700 1 --2- 172.21.15.23:0/2145754691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a100f55a0 0x7f1a10105da0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:41.162 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:41.176+0000 7f1a15593700 1 --2- 172.21.15.23:0/2145754691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a100f6840 0x7f1a10106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:41.163 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:41.176+0000 7f1a15593700 1 -- 172.21.15.23:0/2145754691 >> 172.21.15.23:0/2145754691 conn(0x7f1a100f0ce0 msgr2=0x7f1a100fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:41.163 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:41.176+0000 7f1a15593700 1 -- 172.21.15.23:0/2145754691 shutdown_connections 2021-11-19T08:43:41.163 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:41.176+0000 7f1a15593700 1 -- 172.21.15.23:0/2145754691 wait complete. 2021-11-19T08:43:41.163 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 67 2021-11-19T08:43:41.172 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:34.283331+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:resolve', 'state_seq': 10, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:41.172 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:resolve 2021-11-19T08:43:41.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:41 smithi023 conmon[33490]: audit 2021-11-19T08:43:41.175187+0000 mon.smithi023 (mon.0) 2021-11-19T08:43:41.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:41 smithi023 conmon[33490]: 755 : audit [DBG] from='client.? 172.21.15.23:0/2145754691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:41.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:41 smithi134 conmon[36363]: audit 2021-11-19T08:43:41.175187+0000 mon.smithi023 (mon.0) 755 : audit [DBG] from='client.? 172.21.15.23:0/2145754691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:41.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:41 smithi134 conmon[36363]: 2021-11-19T08:43:42.173 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:42.308 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.321+0000 7f74e2ead700 1 -- 172.21.15.23:0/1502256372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74dc0f5570 msgr2=0x7f74dc0f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:42.308 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.321+0000 7f74e2ead700 1 --2- 172.21.15.23:0/1502256372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74dc0f5570 0x7f74dc0f5990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f74d0004660 tx=0x7f74d0009b30).stop 2021-11-19T08:43:42.308 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.321+0000 7f74e2ead700 1 -- 172.21.15.23:0/1502256372 shutdown_connections 2021-11-19T08:43:42.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.321+0000 7f74e2ead700 1 --2- 172.21.15.23:0/1502256372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74dc0f6810 0x7f74dc0f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:42.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.321+0000 7f74e2ead700 1 --2- 172.21.15.23:0/1502256372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74dc0f5570 0x7f74dc0f5990 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:42.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.321+0000 7f74e2ead700 1 -- 172.21.15.23:0/1502256372 >> 172.21.15.23:0/1502256372 conn(0x7f74dc0f0cb0 msgr2=0x7f74dc0f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:42.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.321+0000 7f74e2ead700 1 -- 172.21.15.23:0/1502256372 shutdown_connections 2021-11-19T08:43:42.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.321+0000 7f74e2ead700 1 -- 172.21.15.23:0/1502256372 wait complete. 2021-11-19T08:43:42.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.321+0000 7f74e2ead700 1 Processor -- start 2021-11-19T08:43:42.310 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.322+0000 7f74e2ead700 1 -- start start 2021-11-19T08:43:42.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.322+0000 7f74e2ead700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74dc0f5570 0x7f74dc060590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:42.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.322+0000 7f74e2ead700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74dc0f6810 0x7f74dc10bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:42.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.322+0000 7f74e2ead700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc10c190 con 0x7f74dc0f6810 2021-11-19T08:43:42.311 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.322+0000 7f74e2ead700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc10c2d0 con 0x7f74dc0f5570 2021-11-19T08:43:42.312 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.322+0000 7f74dbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74dc0f6810 0x7f74dc10bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:42.312 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.322+0000 7f74dbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74dc0f6810 0x7f74dc10bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48030/0 (socket says 172.21.15.23:48030) 2021-11-19T08:43:42.312 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.322+0000 7f74dbfff700 1 -- 172.21.15.23:0/1493283772 learned_addr learned my addr 172.21.15.23:0/1493283772 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:42.312 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.322+0000 7f74e0c49700 1 --2- 172.21.15.23:0/1493283772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74dc0f5570 0x7f74dc060590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:42.313 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.322+0000 7f74dbfff700 1 -- 172.21.15.23:0/1493283772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74dc0f5570 msgr2=0x7f74dc060590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:42.313 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.322+0000 7f74dbfff700 1 --2- 172.21.15.23:0/1493283772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74dc0f5570 0x7f74dc060590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:42.313 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.322+0000 7f74dbfff700 1 -- 172.21.15.23:0/1493283772 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d0005040 con 0x7f74dc0f6810 2021-11-19T08:43:42.314 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.322+0000 7f74dbfff700 1 --2- 172.21.15.23:0/1493283772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74dc0f6810 0x7f74dc10bca0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f74c8002700 tx=0x7f74c8009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:42.314 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.323+0000 7f74d97fa700 1 -- 172.21.15.23:0/1493283772 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74c8009320 con 0x7f74dc0f6810 2021-11-19T08:43:42.314 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.323+0000 7f74d97fa700 1 -- 172.21.15.23:0/1493283772 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74c800f040 con 0x7f74dc0f6810 2021-11-19T08:43:42.314 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.323+0000 7f74d97fa700 1 -- 172.21.15.23:0/1493283772 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74c80147e0 con 0x7f74dc0f6810 2021-11-19T08:43:42.314 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.323+0000 7f74e2ead700 1 -- 172.21.15.23:0/1493283772 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74dc10c530 con 0x7f74dc0f6810 2021-11-19T08:43:42.315 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.323+0000 7f74e2ead700 1 -- 172.21.15.23:0/1493283772 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74dc10caf0 con 0x7f74dc0f6810 2021-11-19T08:43:42.315 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.324+0000 7f74e2ead700 1 -- 172.21.15.23:0/1493283772 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74dc027a00 con 0x7f74dc0f6810 2021-11-19T08:43:42.316 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.324+0000 7f74d97fa700 1 -- 172.21.15.23:0/1493283772 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f74c8009480 con 0x7f74dc0f6810 2021-11-19T08:43:42.317 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.324+0000 7f74d97fa700 1 --2- 172.21.15.23:0/1493283772 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f74cc05c2a0 0x7f74cc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:42.317 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.325+0000 7f74d97fa700 1 -- 172.21.15.23:0/1493283772 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f74c80767d0 con 0x7f74dc0f6810 2021-11-19T08:43:42.317 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.325+0000 7f74e0c49700 1 --2- 172.21.15.23:0/1493283772 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f74cc05c2a0 0x7f74cc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:42.318 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.325+0000 7f74e0c49700 1 --2- 172.21.15.23:0/1493283772 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f74cc05c2a0 0x7f74cc05e760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f74d0009f50 tx=0x7f74d0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:42.318 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.328+0000 7f74d97fa700 1 -- 172.21.15.23:0/1493283772 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74c8049160 con 0x7f74dc0f6810 2021-11-19T08:43:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.525+0000 7f74e2ead700 1 -- 172.21.15.23:0/1493283772 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74dc0fdd00 con 0x7f74dc0f6810 2021-11-19T08:43:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.525+0000 7f74d97fa700 1 -- 172.21.15.23:0/1493283772 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f74c804c780 con 0x7f74dc0f6810 2021-11-19T08:43:42.511 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:42.511 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:34.283331+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:resolve","state_seq":10,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:42.512 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.527+0000 7f74e2ead700 1 -- 172.21.15.23:0/1493283772 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f74cc05c2a0 msgr2=0x7f74cc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:42.513 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.527+0000 7f74e2ead700 1 --2- 172.21.15.23:0/1493283772 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f74cc05c2a0 0x7f74cc05e760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f74d0009f50 tx=0x7f74d0009b30).stop 2021-11-19T08:43:42.513 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.527+0000 7f74e2ead700 1 -- 172.21.15.23:0/1493283772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74dc0f6810 msgr2=0x7f74dc10bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:42.513 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.527+0000 7f74e2ead700 1 --2- 172.21.15.23:0/1493283772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74dc0f6810 0x7f74dc10bca0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f74c8002700 tx=0x7f74c8009de0).stop 2021-11-19T08:43:42.513 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.527+0000 7f74e2ead700 1 -- 172.21.15.23:0/1493283772 shutdown_connections 2021-11-19T08:43:42.514 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.527+0000 7f74e2ead700 1 --2- 172.21.15.23:0/1493283772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74dc0f5570 0x7f74dc060590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:42.514 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.527+0000 7f74e2ead700 1 --2- 172.21.15.23:0/1493283772 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f74cc05c2a0 0x7f74cc05e760 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:42.514 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.527+0000 7f74e2ead700 1 --2- 172.21.15.23:0/1493283772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74dc0f6810 0x7f74dc10bca0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:42.514 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.527+0000 7f74e2ead700 1 -- 172.21.15.23:0/1493283772 >> 172.21.15.23:0/1493283772 conn(0x7f74dc0f0cb0 msgr2=0x7f74dc0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.527+0000 7f74e2ead700 1 -- 172.21.15.23:0/1493283772 shutdown_connections 2021-11-19T08:43:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:42.527+0000 7f74e2ead700 1 -- 172.21.15.23:0/1493283772 wait complete. 2021-11-19T08:43:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 67 2021-11-19T08:43:42.522 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:42 smithi023 conmon[33490]: cluster 2021-11-19T08:43:40.877981+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:43:42.522 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:42 smithi023 conmon[33490]: .14164) 290 : cluster [DBG] pgmap v262: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:42.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:34.283331+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:resolve', 'state_seq': 10, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:42.524 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:resolve 2021-11-19T08:43:42.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:42 smithi134 conmon[36363]: cluster 2021-11-19T08:43:40.877981+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:43:42.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:42 smithi134 conmon[36363]: 290 : cluster [DBG] pgmap v262: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:43.525 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:43.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:43 smithi023 conmon[33490]: audit 2021-11-19T08:43:42.526412+0000 mon.smithi023 (mon 2021-11-19T08:43:43.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:43 smithi023 conmon[33490]: .0) 756 : audit [DBG] from='client.? 172.21.15.23:0/1493283772' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:43.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:43 smithi023 conmon[33490]: audit 2021-11-19T08:43:43.314050+0000 mon.smithi023 (mon.0) 757 : audit [DBG] 2021-11-19T08:43:43.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:43 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:43:43.647 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.661+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1041187932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0580f6820 msgr2=0x7fe0580f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:43.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.661+0000 7fe05fa0e700 1 --2- 172.21.15.23:0/1041187932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0580f6820 0x7fe0580f6c80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fe04c004660 tx=0x7fe04c009b30).stop 2021-11-19T08:43:43.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.661+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1041187932 shutdown_connections 2021-11-19T08:43:43.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.661+0000 7fe05fa0e700 1 --2- 172.21.15.23:0/1041187932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0580f6820 0x7fe0580f6c80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:43.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.661+0000 7fe05fa0e700 1 --2- 172.21.15.23:0/1041187932 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0580f5580 0x7fe0580f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:43.648 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.661+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1041187932 >> 172.21.15.23:0/1041187932 conn(0x7fe0580f0ce0 msgr2=0x7fe0580f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:43.649 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.661+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1041187932 shutdown_connections 2021-11-19T08:43:43.649 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.661+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1041187932 wait complete. 2021-11-19T08:43:43.649 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.662+0000 7fe05fa0e700 1 Processor -- start 2021-11-19T08:43:43.649 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.662+0000 7fe05fa0e700 1 -- start start 2021-11-19T08:43:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.662+0000 7fe05fa0e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0580f5580 0x7fe058060530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.662+0000 7fe05fa0e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0580f6820 0x7fe05810bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.662+0000 7fe05fa0e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05810c0e0 con 0x7fe0580f6820 2021-11-19T08:43:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.662+0000 7fe05fa0e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05810c220 con 0x7fe0580f5580 2021-11-19T08:43:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.662+0000 7fe05cfa9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0580f6820 0x7fe05810bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.662+0000 7fe05cfa9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0580f6820 0x7fe05810bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48040/0 (socket says 172.21.15.23:48040) 2021-11-19T08:43:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.662+0000 7fe05cfa9700 1 -- 172.21.15.23:0/1149339892 learned_addr learned my addr 172.21.15.23:0/1149339892 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.662+0000 7fe05d7aa700 1 --2- 172.21.15.23:0/1149339892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0580f5580 0x7fe058060530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.663+0000 7fe05cfa9700 1 -- 172.21.15.23:0/1149339892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0580f5580 msgr2=0x7fe058060530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:43.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.663+0000 7fe05cfa9700 1 --2- 172.21.15.23:0/1149339892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0580f5580 0x7fe058060530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:43.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.663+0000 7fe05cfa9700 1 -- 172.21.15.23:0/1149339892 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c005040 con 0x7fe0580f6820 2021-11-19T08:43:43.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.663+0000 7fe05d7aa700 1 --2- 172.21.15.23:0/1149339892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0580f5580 0x7fe058060530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-11-19T08:43:43.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.663+0000 7fe05cfa9700 1 --2- 172.21.15.23:0/1149339892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0580f6820 0x7fe05810bbf0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fe04c000c00 tx=0x7fe04c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:43.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.663+0000 7fe055ffb700 1 -- 172.21.15.23:0/1149339892 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe04c019070 con 0x7fe0580f6820 2021-11-19T08:43:43.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.663+0000 7fe055ffb700 1 -- 172.21.15.23:0/1149339892 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe04c007e70 con 0x7fe0580f6820 2021-11-19T08:43:43.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.663+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1149339892 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe05810c450 con 0x7fe0580f6820 2021-11-19T08:43:43.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.663+0000 7fe055ffb700 1 -- 172.21.15.23:0/1149339892 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe04c014c90 con 0x7fe0580f6820 2021-11-19T08:43:43.653 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.663+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1149339892 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05810c9b0 con 0x7fe0580f6820 2021-11-19T08:43:43.654 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.664+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1149339892 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe058027a00 con 0x7fe0580f6820 2021-11-19T08:43:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.665+0000 7fe055ffb700 1 -- 172.21.15.23:0/1149339892 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe04c025070 con 0x7fe0580f6820 2021-11-19T08:43:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.665+0000 7fe055ffb700 1 --2- 172.21.15.23:0/1149339892 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe04005c2a0 0x7fe04005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.665+0000 7fe055ffb700 1 -- 172.21.15.23:0/1149339892 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe04c077350 con 0x7fe0580f6820 2021-11-19T08:43:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.665+0000 7fe05d7aa700 1 --2- 172.21.15.23:0/1149339892 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe04005c2a0 0x7fe04005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.666+0000 7fe05d7aa700 1 --2- 172.21.15.23:0/1149339892 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe04005c2a0 0x7fe04005e760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fe050004fd0 tx=0x7fe05000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.669+0000 7fe055ffb700 1 -- 172.21.15.23:0/1149339892 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe04c049ba0 con 0x7fe0580f6820 2021-11-19T08:43:43.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:43 smithi134 conmon[36363]: audit 2021-11-19T08:43:42.526412+0000 mon.smithi023 (mon.0) 756 : audit [DBG] from='client.? 172.21.15.23:0/1493283772' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:43.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:43 smithi134 conmon[36363]: audit 2021-11-19T08:43:43. 2021-11-19T08:43:43.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:43 smithi134 conmon[36363]: 314050+0000 mon.smithi023 (mon.0) 757 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:43:43.865 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.881+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1149339892 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0580fdd10 con 0x7fe0580f6820 2021-11-19T08:43:43.866 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.882+0000 7fe055ffb700 1 -- 172.21.15.23:0/1149339892 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2575 (secure 0 0 0) 0x7fe04c04d1c0 con 0x7fe0580f6820 2021-11-19T08:43:43.868 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:43.868 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:34.283331+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:resolve","state_seq":10,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:43.869 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.883+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1149339892 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe04005c2a0 msgr2=0x7fe04005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:43.869 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.883+0000 7fe05fa0e700 1 --2- 172.21.15.23:0/1149339892 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe04005c2a0 0x7fe04005e760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fe050004fd0 tx=0x7fe05000b040).stop 2021-11-19T08:43:43.870 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.883+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1149339892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0580f6820 msgr2=0x7fe05810bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:43.870 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.883+0000 7fe05fa0e700 1 --2- 172.21.15.23:0/1149339892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0580f6820 0x7fe05810bbf0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fe04c000c00 tx=0x7fe04c00b040).stop 2021-11-19T08:43:43.870 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.883+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1149339892 shutdown_connections 2021-11-19T08:43:43.870 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.883+0000 7fe05fa0e700 1 --2- 172.21.15.23:0/1149339892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0580f5580 0x7fe058060530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:43.871 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.883+0000 7fe05fa0e700 1 --2- 172.21.15.23:0/1149339892 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe04005c2a0 0x7fe04005e760 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:43.871 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.883+0000 7fe05fa0e700 1 --2- 172.21.15.23:0/1149339892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0580f6820 0x7fe05810bbf0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:43.871 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.883+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1149339892 >> 172.21.15.23:0/1149339892 conn(0x7fe0580f0ce0 msgr2=0x7fe0580fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:43.871 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.883+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1149339892 shutdown_connections 2021-11-19T08:43:43.871 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:43.883+0000 7fe05fa0e700 1 -- 172.21.15.23:0/1149339892 wait complete. 2021-11-19T08:43:43.872 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 67 2021-11-19T08:43:43.880 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:34.283331+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:resolve', 'state_seq': 10, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:43.880 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:resolve 2021-11-19T08:43:44.882 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:45.017 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.030+0000 7fa6e6253700 1 -- 172.21.15.23:0/2304676422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e00f6770 msgr2=0x7fa6e00f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:45.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.030+0000 7fa6e6253700 1 --2- 172.21.15.23:0/2304676422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e00f6770 0x7fa6e00f6bd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa6d0004660 tx=0x7fa6d0009b30).stop 2021-11-19T08:43:45.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.030+0000 7fa6e6253700 1 -- 172.21.15.23:0/2304676422 shutdown_connections 2021-11-19T08:43:45.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.030+0000 7fa6e6253700 1 --2- 172.21.15.23:0/2304676422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e00f6770 0x7fa6e00f6bd0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:45.018 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.030+0000 7fa6e6253700 1 --2- 172.21.15.23:0/2304676422 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6e00f54d0 0x7fa6e00f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.030+0000 7fa6e6253700 1 -- 172.21.15.23:0/2304676422 >> 172.21.15.23:0/2304676422 conn(0x7fa6e00f0c10 msgr2=0x7fa6e00f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.031+0000 7fa6e6253700 1 -- 172.21.15.23:0/2304676422 shutdown_connections 2021-11-19T08:43:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.031+0000 7fa6e6253700 1 -- 172.21.15.23:0/2304676422 wait complete. 2021-11-19T08:43:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.031+0000 7fa6e6253700 1 Processor -- start 2021-11-19T08:43:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.031+0000 7fa6e6253700 1 -- start start 2021-11-19T08:43:45.020 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.031+0000 7fa6e6253700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6e00f54d0 0x7fa6e0106d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:45.020 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.031+0000 7fa6e6253700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e00f6770 0x7fa6e0107230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:45.020 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.031+0000 7fa6e6253700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e01077e0 con 0x7fa6e00f6770 2021-11-19T08:43:45.020 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.031+0000 7fa6e6253700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e0107fe0 con 0x7fa6e00f54d0 2021-11-19T08:43:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.031+0000 7fa6e4a50700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e00f6770 0x7fa6e0107230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.031+0000 7fa6e4a50700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e00f6770 0x7fa6e0107230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48050/0 (socket says 172.21.15.23:48050) 2021-11-19T08:43:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.031+0000 7fa6e4a50700 1 -- 172.21.15.23:0/4009297071 learned_addr learned my addr 172.21.15.23:0/4009297071 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.032+0000 7fa6e5251700 1 --2- 172.21.15.23:0/4009297071 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6e00f54d0 0x7fa6e0106d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.032+0000 7fa6e4a50700 1 -- 172.21.15.23:0/4009297071 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6e00f54d0 msgr2=0x7fa6e0106d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.032+0000 7fa6e4a50700 1 --2- 172.21.15.23:0/4009297071 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6e00f54d0 0x7fa6e0106d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.032+0000 7fa6e4a50700 1 -- 172.21.15.23:0/4009297071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d0005040 con 0x7fa6e00f6770 2021-11-19T08:43:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.032+0000 7fa6e5251700 1 --2- 172.21.15.23:0/4009297071 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6e00f54d0 0x7fa6e0106d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:43:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.032+0000 7fa6e4a50700 1 --2- 172.21.15.23:0/4009297071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e00f6770 0x7fa6e0107230 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fa6d0009f50 tx=0x7fa6d0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:45.023 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.032+0000 7fa6d5ffb700 1 -- 172.21.15.23:0/4009297071 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6d0018070 con 0x7fa6e00f6770 2021-11-19T08:43:45.023 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.032+0000 7fa6d5ffb700 1 -- 172.21.15.23:0/4009297071 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6d0013410 con 0x7fa6e00f6770 2021-11-19T08:43:45.023 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.032+0000 7fa6e6253700 1 -- 172.21.15.23:0/4009297071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e0108260 con 0x7fa6e00f6770 2021-11-19T08:43:45.023 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.032+0000 7fa6d5ffb700 1 -- 172.21.15.23:0/4009297071 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6d001c3c0 con 0x7fa6e00f6770 2021-11-19T08:43:45.024 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.032+0000 7fa6e6253700 1 -- 172.21.15.23:0/4009297071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e01087c0 con 0x7fa6e00f6770 2021-11-19T08:43:45.024 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.033+0000 7fa6e6253700 1 -- 172.21.15.23:0/4009297071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e00f48c0 con 0x7fa6e00f6770 2021-11-19T08:43:45.025 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.034+0000 7fa6d5ffb700 1 -- 172.21.15.23:0/4009297071 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa6d001d420 con 0x7fa6e00f6770 2021-11-19T08:43:45.025 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.034+0000 7fa6d5ffb700 1 --2- 172.21.15.23:0/4009297071 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa6cc05c300 0x7fa6cc05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:45.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.034+0000 7fa6d5ffb700 1 -- 172.21.15.23:0/4009297071 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa6d0077f70 con 0x7fa6e00f6770 2021-11-19T08:43:45.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.034+0000 7fa6e5251700 1 --2- 172.21.15.23:0/4009297071 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa6cc05c300 0x7fa6cc05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:45.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.034+0000 7fa6e5251700 1 --2- 172.21.15.23:0/4009297071 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa6cc05c300 0x7fa6cc05e7c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa6e00f65d0 tx=0x7fa6dc009c80).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:45.026 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.038+0000 7fa6d5ffb700 1 -- 172.21.15.23:0/4009297071 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6d00a42b0 con 0x7fa6e00f6770 2021-11-19T08:43:45.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:44 smithi134 conmon[36363]: cluster 2021-11-19T08:43:42.878340+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:43:45.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:44 smithi134 conmon[36363]: ) 291 : cluster [DBG] pgmap v263: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:45.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:44 smithi134 conmon[36363]: audit 2021-11-19T08:43:43.726734+0000 mon.smithi023 (mon.0) 758 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:43:45.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:44 smithi134 conmon[36363]: audit 2021-11-19T08:43:43.882757+0000 mon.smithi023 (mon.0) 759 : audit [DBG] from='client.? 172.21.15.23:0/1149339892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:45.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:44 smithi023 conmon[33490]: cluster 2021-11-19T08:43:42.878340+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:43:45.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:44 smithi023 conmon[33490]: .14164) 291 : cluster [DBG] pgmap v263: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:43:45.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:44 smithi023 conmon[33490]: audit 2021-11-19T08:43:43.726734+0000 mon.smithi023 (mon.0) 758 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:43:45.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:44 smithi023 conmon[33490]: audit 2021-11-19T08:43:43.882757+0000 mon.smithi023 (mon.0) 759 : audit [DBG] from='client.? 172.21.15.23:0/1149339892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:45.219 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.234+0000 7fa6e6253700 1 -- 172.21.15.23:0/4009297071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6e0045ad0 con 0x7fa6e00f6770 2021-11-19T08:43:45.219 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.235+0000 7fa6d5ffb700 1 -- 172.21.15.23:0/4009297071 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2575 (secure 0 0 0) 0x7fa6e0045ad0 con 0x7fa6e00f6770 2021-11-19T08:43:45.220 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:45.220 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:34.283331+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:resolve","state_seq":10,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:45.221 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.237+0000 7fa6e6253700 1 -- 172.21.15.23:0/4009297071 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa6cc05c300 msgr2=0x7fa6cc05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:45.222 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.237+0000 7fa6e6253700 1 --2- 172.21.15.23:0/4009297071 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa6cc05c300 0x7fa6cc05e7c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa6e00f65d0 tx=0x7fa6dc009c80).stop 2021-11-19T08:43:45.222 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.237+0000 7fa6e6253700 1 -- 172.21.15.23:0/4009297071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e00f6770 msgr2=0x7fa6e0107230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:45.222 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.237+0000 7fa6e6253700 1 --2- 172.21.15.23:0/4009297071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e00f6770 0x7fa6e0107230 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fa6d0009f50 tx=0x7fa6d0009b30).stop 2021-11-19T08:43:45.222 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.237+0000 7fa6e6253700 1 -- 172.21.15.23:0/4009297071 shutdown_connections 2021-11-19T08:43:45.223 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.237+0000 7fa6e6253700 1 --2- 172.21.15.23:0/4009297071 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6e00f54d0 0x7fa6e0106d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:45.223 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.237+0000 7fa6e6253700 1 --2- 172.21.15.23:0/4009297071 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa6cc05c300 0x7fa6cc05e7c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:45.223 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.237+0000 7fa6e6253700 1 --2- 172.21.15.23:0/4009297071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e00f6770 0x7fa6e0107230 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:45.223 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.237+0000 7fa6e6253700 1 -- 172.21.15.23:0/4009297071 >> 172.21.15.23:0/4009297071 conn(0x7fa6e00f0c10 msgr2=0x7fa6e005f760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:45.224 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.237+0000 7fa6e6253700 1 -- 172.21.15.23:0/4009297071 shutdown_connections 2021-11-19T08:43:45.224 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:45.237+0000 7fa6e6253700 1 -- 172.21.15.23:0/4009297071 wait complete. 2021-11-19T08:43:45.224 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 67 2021-11-19T08:43:45.233 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:34.283331+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:resolve', 'state_seq': 10, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:45.233 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:resolve 2021-11-19T08:43:46.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:45 smithi134 conmon[36363]: audit 2021-11-19T08:43:45.235879+0000 mon.smithi023 (mon.0) 760 2021-11-19T08:43:46.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:45 smithi134 conmon[36363]: : audit [DBG] from='client.? 172.21.15.23:0/4009297071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:46.112 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:45 smithi023 conmon[33490]: audit 2021-11-19T08:43:45.235879+0000 mon.smithi023 (mon.0) 760 : audit [DBG] from='client.? 172.21.15.23:0/4009297071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:46.234 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.382+0000 7f49a96f0700 1 -- 172.21.15.23:0/3581203175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a40f5870 msgr2=0x7f49a40f5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:46.371 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.382+0000 7f49a96f0700 1 --2- 172.21.15.23:0/3581203175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a40f5870 0x7f49a40f5cf0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f4994004660 tx=0x7f4994009b30).stop 2021-11-19T08:43:46.371 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.382+0000 7f49a96f0700 1 -- 172.21.15.23:0/3581203175 shutdown_connections 2021-11-19T08:43:46.371 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.382+0000 7f49a96f0700 1 --2- 172.21.15.23:0/3581203175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a40f5870 0x7f49a40f5cf0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:46.371 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.382+0000 7f49a96f0700 1 --2- 172.21.15.23:0/3581203175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49a40f6b70 0x7f49a40f6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:46.372 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.382+0000 7f49a96f0700 1 -- 172.21.15.23:0/3581203175 >> 172.21.15.23:0/3581203175 conn(0x7f49a40f0440 msgr2=0x7f49a40f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:46.372 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.382+0000 7f49a96f0700 1 -- 172.21.15.23:0/3581203175 shutdown_connections 2021-11-19T08:43:46.372 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.382+0000 7f49a96f0700 1 -- 172.21.15.23:0/3581203175 wait complete. 2021-11-19T08:43:46.372 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.383+0000 7f49a96f0700 1 Processor -- start 2021-11-19T08:43:46.372 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.383+0000 7f49a96f0700 1 -- start start 2021-11-19T08:43:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.383+0000 7f49a96f0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a40f5870 0x7f49a40f5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.383+0000 7f49a96f0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49a40f6b70 0x7f49a410bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.383+0000 7f49a96f0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a410c1a0 con 0x7f49a40f5870 2021-11-19T08:43:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.383+0000 7f49a96f0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a410c2e0 con 0x7f49a40f6b70 2021-11-19T08:43:46.375 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.383+0000 7f49a2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a40f5870 0x7f49a40f5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:46.375 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.383+0000 7f49a2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a40f5870 0x7f49a40f5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48058/0 (socket says 172.21.15.23:48058) 2021-11-19T08:43:46.376 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.383+0000 7f49a2ffd700 1 -- 172.21.15.23:0/2501669495 learned_addr learned my addr 172.21.15.23:0/2501669495 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:46.376 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.383+0000 7f49a27fc700 1 --2- 172.21.15.23:0/2501669495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49a40f6b70 0x7f49a410bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:46.377 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.384+0000 7f49a2ffd700 1 -- 172.21.15.23:0/2501669495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49a40f6b70 msgr2=0x7f49a410bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:46.377 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.384+0000 7f49a2ffd700 1 --2- 172.21.15.23:0/2501669495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49a40f6b70 0x7f49a410bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:46.378 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.384+0000 7f49a2ffd700 1 -- 172.21.15.23:0/2501669495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4994005040 con 0x7f49a40f5870 2021-11-19T08:43:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.384+0000 7f49a2ffd700 1 --2- 172.21.15.23:0/2501669495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a40f5870 0x7f49a40f5230 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f4998002730 tx=0x7f4998009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.384+0000 7f498b7fe700 1 -- 172.21.15.23:0/2501669495 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49980079c0 con 0x7f49a40f5870 2021-11-19T08:43:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.384+0000 7f49a96f0700 1 -- 172.21.15.23:0/2501669495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a410c540 con 0x7f49a40f5870 2021-11-19T08:43:46.380 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.384+0000 7f498b7fe700 1 -- 172.21.15.23:0/2501669495 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f499800f040 con 0x7f49a40f5870 2021-11-19T08:43:46.380 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.384+0000 7f498b7fe700 1 -- 172.21.15.23:0/2501669495 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4998014870 con 0x7f49a40f5870 2021-11-19T08:43:46.381 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.384+0000 7f49a96f0700 1 -- 172.21.15.23:0/2501669495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a410cb00 con 0x7f49a40f5870 2021-11-19T08:43:46.381 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.385+0000 7f49a96f0700 1 -- 172.21.15.23:0/2501669495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a4027a00 con 0x7f49a40f5870 2021-11-19T08:43:46.382 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.385+0000 7f498b7fe700 1 -- 172.21.15.23:0/2501669495 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f499801b060 con 0x7f49a40f5870 2021-11-19T08:43:46.382 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.386+0000 7f498b7fe700 1 --2- 172.21.15.23:0/2501669495 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f498c05c2a0 0x7f498c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:46.382 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.386+0000 7f498b7fe700 1 -- 172.21.15.23:0/2501669495 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f49980760c0 con 0x7f49a40f5870 2021-11-19T08:43:46.383 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.386+0000 7f49a27fc700 1 --2- 172.21.15.23:0/2501669495 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f498c05c2a0 0x7f498c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:46.383 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.386+0000 7f49a27fc700 1 --2- 172.21.15.23:0/2501669495 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f498c05c2a0 0x7f498c05e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f4994009f50 tx=0x7f4994009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:46.385 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.390+0000 7f498b7fe700 1 -- 172.21.15.23:0/2501669495 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49980489d0 con 0x7f49a40f5870 2021-11-19T08:43:46.569 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.584+0000 7f49a96f0700 1 -- 172.21.15.23:0/2501669495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49a40fdd60 con 0x7f49a40f5870 2021-11-19T08:43:46.569 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.585+0000 7f498b7fe700 1 -- 172.21.15.23:0/2501669495 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f499804bff0 con 0x7f49a40f5870 2021-11-19T08:43:46.570 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:46.570 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:34.283331+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:resolve","state_seq":10,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:46.571 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.586+0000 7f49a96f0700 1 -- 172.21.15.23:0/2501669495 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f498c05c2a0 msgr2=0x7f498c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:46.572 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.587+0000 7f49a96f0700 1 --2- 172.21.15.23:0/2501669495 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f498c05c2a0 0x7f498c05e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f4994009f50 tx=0x7f4994009b30).stop 2021-11-19T08:43:46.572 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.587+0000 7f49a96f0700 1 -- 172.21.15.23:0/2501669495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a40f5870 msgr2=0x7f49a40f5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:46.573 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.587+0000 7f49a96f0700 1 --2- 172.21.15.23:0/2501669495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a40f5870 0x7f49a40f5230 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f4998002730 tx=0x7f4998009d80).stop 2021-11-19T08:43:46.573 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.587+0000 7f49a96f0700 1 -- 172.21.15.23:0/2501669495 shutdown_connections 2021-11-19T08:43:46.573 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.587+0000 7f49a96f0700 1 --2- 172.21.15.23:0/2501669495 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f498c05c2a0 0x7f498c05e760 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:46.574 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.587+0000 7f49a96f0700 1 --2- 172.21.15.23:0/2501669495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49a40f5870 0x7f49a40f5230 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:46.574 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.587+0000 7f49a96f0700 1 --2- 172.21.15.23:0/2501669495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f49a40f6b70 0x7f49a410bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:46.574 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.587+0000 7f49a96f0700 1 -- 172.21.15.23:0/2501669495 >> 172.21.15.23:0/2501669495 conn(0x7f49a40f0440 msgr2=0x7f49a40fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:46.574 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.587+0000 7f49a96f0700 1 -- 172.21.15.23:0/2501669495 shutdown_connections 2021-11-19T08:43:46.575 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:46.587+0000 7f49a96f0700 1 -- 172.21.15.23:0/2501669495 wait complete. 2021-11-19T08:43:46.575 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 67 2021-11-19T08:43:46.583 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:34.283331+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:resolve', 'state_seq': 10, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:46.584 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:resolve 2021-11-19T08:43:46.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:46 smithi023 conmon[33490]: cluster 2021-11-19T08:43:44 2021-11-19T08:43:46.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:46 smithi023 conmon[33490]: .878802+0000 mgr.smithi023.ehlpcl (mgr.14164) 292 : cluster [DBG] pgmap v264: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:43:46.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:46 smithi023 conmon[33490]: audit 2021-11-19T08:43:46. 2021-11-19T08:43:46.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:46 smithi023 conmon[33490]: 586037+0000 mon.smithi023 (mon.0) 761 : audit [DBG] from='client.? 172.21.15.23:0/2501669495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:47.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:46 smithi134 conmon[36363]: cluster 2021-11-19T08:43:44.878802+0000 mgr.smithi023.ehlpcl (mgr.14164) 292 : cluster [DBG] pgmap v264: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:43:47.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:46 smithi134 conmon[36363]: audit 2021-11-19T08:43:46.586037+0000 mon.smithi023 (mon.0) 761 : audit [DBG] from='client.? 172.21.15.23:0/2501669495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:47.585 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:47.721 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.732+0000 7f0f34de1700 1 -- 172.21.15.23:0/2011207640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f300f5580 msgr2=0x7f0f300f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:47.722 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.732+0000 7f0f34de1700 1 --2- 172.21.15.23:0/2011207640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f300f5580 0x7f0f300f59a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f0f20004660 tx=0x7f0f20009b30).stop 2021-11-19T08:43:47.722 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f34de1700 1 -- 172.21.15.23:0/2011207640 shutdown_connections 2021-11-19T08:43:47.722 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f34de1700 1 --2- 172.21.15.23:0/2011207640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f300f6820 0x7f0f300f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:47.722 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f34de1700 1 --2- 172.21.15.23:0/2011207640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f300f5580 0x7f0f300f59a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:47.723 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f34de1700 1 -- 172.21.15.23:0/2011207640 >> 172.21.15.23:0/2011207640 conn(0x7f0f300f0ce0 msgr2=0x7f0f300f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:47.723 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f34de1700 1 -- 172.21.15.23:0/2011207640 shutdown_connections 2021-11-19T08:43:47.723 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f34de1700 1 -- 172.21.15.23:0/2011207640 wait complete. 2021-11-19T08:43:47.723 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f34de1700 1 Processor -- start 2021-11-19T08:43:47.723 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f34de1700 1 -- start start 2021-11-19T08:43:47.724 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f34de1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f300f5580 0x7f0f300604d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:47.724 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f34de1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f300f6820 0x7f0f3010e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:47.724 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f34de1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f30107a80 con 0x7f0f300f5580 2021-11-19T08:43:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f34de1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3010e590 con 0x7f0f300f6820 2021-11-19T08:43:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f2e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f300f5580 0x7f0f300604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f2e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f300f5580 0x7f0f300604d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48068/0 (socket says 172.21.15.23:48068) 2021-11-19T08:43:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.733+0000 7f0f2e59c700 1 -- 172.21.15.23:0/3198102583 learned_addr learned my addr 172.21.15.23:0/3198102583 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.734+0000 7f0f2dd9b700 1 --2- 172.21.15.23:0/3198102583 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f300f6820 0x7f0f3010e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.734+0000 7f0f2e59c700 1 -- 172.21.15.23:0/3198102583 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f300f6820 msgr2=0x7f0f3010e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.734+0000 7f0f2e59c700 1 --2- 172.21.15.23:0/3198102583 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f300f6820 0x7f0f3010e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.734+0000 7f0f2e59c700 1 -- 172.21.15.23:0/3198102583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f20005040 con 0x7f0f300f5580 2021-11-19T08:43:47.727 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.734+0000 7f0f2dd9b700 1 --2- 172.21.15.23:0/3198102583 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f300f6820 0x7f0f3010e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:43:47.727 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.734+0000 7f0f2e59c700 1 --2- 172.21.15.23:0/3198102583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f300f5580 0x7f0f300604d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f0f20007aa0 tx=0x7f0f20009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:47.728 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.734+0000 7f0f1effd700 1 -- 172.21.15.23:0/3198102583 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f20018070 con 0x7f0f300f5580 2021-11-19T08:43:47.728 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.734+0000 7f0f1effd700 1 -- 172.21.15.23:0/3198102583 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f20009700 con 0x7f0f300f5580 2021-11-19T08:43:47.728 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.734+0000 7f0f34de1700 1 -- 172.21.15.23:0/3198102583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f3010e6d0 con 0x7f0f300f5580 2021-11-19T08:43:47.729 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.734+0000 7f0f1effd700 1 -- 172.21.15.23:0/3198102583 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f2001c620 con 0x7f0f300f5580 2021-11-19T08:43:47.729 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.734+0000 7f0f34de1700 1 -- 172.21.15.23:0/3198102583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3010ec30 con 0x7f0f300f5580 2021-11-19T08:43:47.729 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.735+0000 7f0f34de1700 1 -- 172.21.15.23:0/3198102583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f30027a00 con 0x7f0f300f5580 2021-11-19T08:43:47.729 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.736+0000 7f0f1effd700 1 -- 172.21.15.23:0/3198102583 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0f2001d420 con 0x7f0f300f5580 2021-11-19T08:43:47.730 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.736+0000 7f0f1effd700 1 --2- 172.21.15.23:0/3198102583 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f1805c2a0 0x7f0f1805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:47.730 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.736+0000 7f0f1effd700 1 -- 172.21.15.23:0/3198102583 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0f20078220 con 0x7f0f300f5580 2021-11-19T08:43:47.731 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.736+0000 7f0f2dd9b700 1 --2- 172.21.15.23:0/3198102583 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f1805c2a0 0x7f0f1805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:47.731 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.736+0000 7f0f2dd9b700 1 --2- 172.21.15.23:0/3198102583 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f1805c2a0 0x7f0f1805e760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f0f240041d0 tx=0x7f0f24009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:47.731 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.740+0000 7f0f1effd700 1 -- 172.21.15.23:0/3198102583 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f2004aa70 con 0x7f0f300f5580 2021-11-19T08:43:47.920 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.934+0000 7f0f34de1700 1 -- 172.21.15.23:0/3198102583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f300fdd10 con 0x7f0f300f5580 2021-11-19T08:43:47.920 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.935+0000 7f0f1effd700 1 -- 172.21.15.23:0/3198102583 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f0f2004e090 con 0x7f0f300f5580 2021-11-19T08:43:47.921 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:47.921 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:34.283331+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":24381},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24381":{"gid":24381,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":59,"state":"up:active","state_seq":21,"addr":"172.21.15.134:6827/348816881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":348816881},{"type":"v1","addr":"172.21.15.134:6827","nonce":348816881}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:resolve","state_seq":10,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:47.922 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.936+0000 7f0f34de1700 1 -- 172.21.15.23:0/3198102583 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f1805c2a0 msgr2=0x7f0f1805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:47.923 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.936+0000 7f0f34de1700 1 --2- 172.21.15.23:0/3198102583 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f1805c2a0 0x7f0f1805e760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f0f240041d0 tx=0x7f0f24009040).stop 2021-11-19T08:43:47.923 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.936+0000 7f0f34de1700 1 -- 172.21.15.23:0/3198102583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f300f5580 msgr2=0x7f0f300604d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:47.923 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.936+0000 7f0f34de1700 1 --2- 172.21.15.23:0/3198102583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f300f5580 0x7f0f300604d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f0f20007aa0 tx=0x7f0f20009b30).stop 2021-11-19T08:43:47.923 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.936+0000 7f0f34de1700 1 -- 172.21.15.23:0/3198102583 shutdown_connections 2021-11-19T08:43:47.924 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.936+0000 7f0f34de1700 1 --2- 172.21.15.23:0/3198102583 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0f1805c2a0 0x7f0f1805e760 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:47.924 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.936+0000 7f0f34de1700 1 --2- 172.21.15.23:0/3198102583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f300f5580 0x7f0f300604d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:47.924 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.936+0000 7f0f34de1700 1 --2- 172.21.15.23:0/3198102583 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f300f6820 0x7f0f3010e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:47.924 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.936+0000 7f0f34de1700 1 -- 172.21.15.23:0/3198102583 >> 172.21.15.23:0/3198102583 conn(0x7f0f300f0ce0 msgr2=0x7f0f300fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:47.925 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.936+0000 7f0f34de1700 1 -- 172.21.15.23:0/3198102583 shutdown_connections 2021-11-19T08:43:47.925 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:47.936+0000 7f0f34de1700 1 -- 172.21.15.23:0/3198102583 wait complete. 2021-11-19T08:43:47.925 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 67 2021-11-19T08:43:47.934 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:34.283331+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 52, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 24381}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24381': {'gid': 24381, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 59, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.134:6827/348816881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 348816881}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 348816881}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:resolve', 'state_seq': 10, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:47.934 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:resolve 2021-11-19T08:43:48.936 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:49.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:48 smithi134 conmon[36363]: cluster 2021-11-19T08:43:46.879400+0000 mgr.smithi023.ehlpcl (mgr.14164) 293 : cluster [DBG] pgmap v265: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-19T08:43:49.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:48 smithi134 conmon[36363]: audit 2021-11-19T08:43:47.936649+0000 mon.smithi023 (mon.0) 762 : audit [DBG] from='client.? 172.21.15.23:0/3198102583' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:49.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:48 smithi134 conmon[36363]: cluster 2021-11-19T08:43:48.297724+0000 mon.smithi023 (mon.0) 763 : cluster [INF] daemon mds.cephfs.smithi134.opxllb restarted 2021-11-19T08:43:49.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:48 smithi134 conmon[36363]: cluster 2021-11-19T08:43:48.306154 2021-11-19T08:43:49.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:48 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 764 : cluster [DBG] osdmap e53: 6 total, 6 up, 6 in 2021-11-19T08:43:49.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.083+0000 7f46c9a94700 1 -- 172.21.15.23:0/831704087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46c40f6820 msgr2=0x7f46c40f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:49.071 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.083+0000 7f46c9a94700 1 --2- 172.21.15.23:0/831704087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46c40f6820 0x7f46c40f6c80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f46b4004660 tx=0x7f46b4009b30).stop 2021-11-19T08:43:49.072 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.083+0000 7f46c9a94700 1 -- 172.21.15.23:0/831704087 shutdown_connections 2021-11-19T08:43:49.072 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.083+0000 7f46c9a94700 1 --2- 172.21.15.23:0/831704087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46c40f6820 0x7f46c40f6c80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:49.072 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.083+0000 7f46c9a94700 1 --2- 172.21.15.23:0/831704087 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46c40f5580 0x7f46c40f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:49.072 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.084+0000 7f46c9a94700 1 -- 172.21.15.23:0/831704087 >> 172.21.15.23:0/831704087 conn(0x7f46c40f0ce0 msgr2=0x7f46c40f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:49.073 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.084+0000 7f46c9a94700 1 -- 172.21.15.23:0/831704087 shutdown_connections 2021-11-19T08:43:49.073 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.084+0000 7f46c9a94700 1 -- 172.21.15.23:0/831704087 wait complete. 2021-11-19T08:43:49.074 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.084+0000 7f46c9a94700 1 Processor -- start 2021-11-19T08:43:49.074 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.084+0000 7f46c9a94700 1 -- start start 2021-11-19T08:43:49.074 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.084+0000 7f46c9a94700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46c40f5580 0x7f46c4106e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:49.075 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.084+0000 7f46c9a94700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46c40f6820 0x7f46c4107310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:49.075 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.084+0000 7f46c9a94700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c41078c0 con 0x7f46c40f5580 2021-11-19T08:43:49.075 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.084+0000 7f46c9a94700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c4107a00 con 0x7f46c40f6820 2021-11-19T08:43:49.075 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.085+0000 7f46c37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46c40f5580 0x7f46c4106e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:49.075 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.085+0000 7f46c37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46c40f5580 0x7f46c4106e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48088/0 (socket says 172.21.15.23:48088) 2021-11-19T08:43:49.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.085+0000 7f46c37fe700 1 -- 172.21.15.23:0/1730209889 learned_addr learned my addr 172.21.15.23:0/1730209889 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:49.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.085+0000 7f46c2ffd700 1 --2- 172.21.15.23:0/1730209889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46c40f6820 0x7f46c4107310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:49.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.085+0000 7f46c37fe700 1 -- 172.21.15.23:0/1730209889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46c40f6820 msgr2=0x7f46c4107310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:49.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.085+0000 7f46c37fe700 1 --2- 172.21.15.23:0/1730209889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46c40f6820 0x7f46c4107310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:49.077 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.085+0000 7f46c37fe700 1 -- 172.21.15.23:0/1730209889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b4005040 con 0x7f46c40f5580 2021-11-19T08:43:49.077 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.085+0000 7f46c37fe700 1 --2- 172.21.15.23:0/1730209889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46c40f5580 0x7f46c4106e20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f46b8002700 tx=0x7f46b8009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:49.078 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.085+0000 7f46abfff700 1 -- 172.21.15.23:0/1730209889 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46b80079c0 con 0x7f46c40f5580 2021-11-19T08:43:49.078 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.085+0000 7f46c9a94700 1 -- 172.21.15.23:0/1730209889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c410c4a0 con 0x7f46c40f5580 2021-11-19T08:43:49.078 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.085+0000 7f46abfff700 1 -- 172.21.15.23:0/1730209889 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46b800f040 con 0x7f46c40f5580 2021-11-19T08:43:49.078 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.085+0000 7f46abfff700 1 -- 172.21.15.23:0/1730209889 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46b8014850 con 0x7f46c40f5580 2021-11-19T08:43:49.078 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.085+0000 7f46c9a94700 1 -- 172.21.15.23:0/1730209889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c410ca60 con 0x7f46c40f5580 2021-11-19T08:43:49.079 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.086+0000 7f46c9a94700 1 -- 172.21.15.23:0/1730209889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c40fdd10 con 0x7f46c40f5580 2021-11-19T08:43:49.080 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.087+0000 7f46abfff700 1 -- 172.21.15.23:0/1730209889 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f46b801b060 con 0x7f46c40f5580 2021-11-19T08:43:49.080 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.087+0000 7f46abfff700 1 --2- 172.21.15.23:0/1730209889 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f46ac05c2a0 0x7f46ac05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:49.081 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.087+0000 7f46abfff700 1 -- 172.21.15.23:0/1730209889 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f46b8076240 con 0x7f46c40f5580 2021-11-19T08:43:49.081 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.087+0000 7f46c2ffd700 1 --2- 172.21.15.23:0/1730209889 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f46ac05c2a0 0x7f46ac05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:49.081 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.088+0000 7f46c2ffd700 1 --2- 172.21.15.23:0/1730209889 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f46ac05c2a0 0x7f46ac05e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f46b4009f50 tx=0x7f46b4009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:49.081 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.091+0000 7f46abfff700 1 -- 172.21.15.23:0/1730209889 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46b8048aa0 con 0x7f46c40f5580 2021-11-19T08:43:49.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:48 smithi023 conmon[33490]: cluster 2021-11-19T08:43:46.879400+0000 mgr.smithi023.ehlpcl (mgr.14164) 293 : cluster [DBG] pgmap v265: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2021-11-19T08:43:49.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:48 smithi023 conmon[33490]: audit 2021-11-19T08:43:47.936649+0000 mon.smithi023 (mon.0) 762 : audit [DBG] from='client.? 172.21.15.23:0/3198102583' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:49.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:48 smithi023 conmon[33490]: cluster 2021-11-19T08:43:48.297724+0000 mon.smithi023 (mon.0) 763 : cluster [INF] daemon mds.cephfs.smithi134.opxllb restarted 2021-11-19T08:43:49.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:48 smithi023 conmon[33490]: cluster 2021-11-19T08:43:48.306154+0000 mon.smithi023 (mon.0) 764 : cluster [DBG] osdmap e53: 6 total, 6 up, 6 in 2021-11-19T08:43:49.280 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.294+0000 7f46c9a94700 1 -- 172.21.15.23:0/1730209889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46c410d160 con 0x7f46c40f5580 2021-11-19T08:43:49.280 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.294+0000 7f46abfff700 1 -- 172.21.15.23:0/1730209889 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2578 (secure 0 0 0) 0x7f46b804c0c0 con 0x7f46c40f5580 2021-11-19T08:43:49.281 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:49.281 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":69,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:48.751845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:replay","state_seq":2,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:resolve","state_seq":10,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:49.283 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.296+0000 7f46c9a94700 1 -- 172.21.15.23:0/1730209889 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f46ac05c2a0 msgr2=0x7f46ac05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:49.284 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.296+0000 7f46c9a94700 1 --2- 172.21.15.23:0/1730209889 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f46ac05c2a0 0x7f46ac05e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f46b4009f50 tx=0x7f46b4009b30).stop 2021-11-19T08:43:49.284 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.296+0000 7f46c9a94700 1 -- 172.21.15.23:0/1730209889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46c40f5580 msgr2=0x7f46c4106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:49.285 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.296+0000 7f46c9a94700 1 --2- 172.21.15.23:0/1730209889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46c40f5580 0x7f46c4106e20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f46b8002700 tx=0x7f46b8009e00).stop 2021-11-19T08:43:49.285 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.296+0000 7f46c9a94700 1 -- 172.21.15.23:0/1730209889 shutdown_connections 2021-11-19T08:43:49.285 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.296+0000 7f46c9a94700 1 --2- 172.21.15.23:0/1730209889 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f46ac05c2a0 0x7f46ac05e760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:49.286 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.296+0000 7f46c9a94700 1 --2- 172.21.15.23:0/1730209889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46c40f5580 0x7f46c4106e20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:49.286 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.296+0000 7f46c9a94700 1 --2- 172.21.15.23:0/1730209889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46c40f6820 0x7f46c4107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:49.286 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.296+0000 7f46c9a94700 1 -- 172.21.15.23:0/1730209889 >> 172.21.15.23:0/1730209889 conn(0x7f46c40f0ce0 msgr2=0x7f46c4060260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:49.287 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.296+0000 7f46c9a94700 1 -- 172.21.15.23:0/1730209889 shutdown_connections 2021-11-19T08:43:49.287 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:49.296+0000 7f46c9a94700 1 -- 172.21.15.23:0/1730209889 wait complete. 2021-11-19T08:43:49.287 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 69 2021-11-19T08:43:49.293 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:48.751845+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:resolve', 'state_seq': 10, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:49.294 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14732:up:replay 2021-11-19T08:43:50.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:49 smithi134 conmon[36363]: cluster 2021-11-19T08:43:48.751679+0000 mon.smithi023 (mon.0) 765 : cluster [DBG] mds.? [v2:172.21.15.134:6826/3725515320,v1:172.21.15.134:6827/3725515320] up:boot 2021-11-19T08:43:50.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:49 smithi134 conmon[36363]: 2021-11-19T08:43:50.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:49 smithi134 conmon[36363]: cluster 2021-11-19T08:43:48.751789+0000 mon.smithi023 (mon.0) 766 : cluster [INF] Standby daemon mds.cephfs.smithi023.fuvzkr assigned to filesystem cephfs as rank 1 2021-11-19T08:43:50.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:49 smithi134 conmon[36363]: cluster 2021-11-19T08:43:48.754168+0000 mon.smithi023 (mon. 2021-11-19T08:43:50.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:49 smithi134 conmon[36363]: 0) 767 : cluster [DBG] fsmap cephfs:1/2 {0=cephfs.smithi134.jqsnzo=up:resolve} 2 up:standby, 1 failed 2021-11-19T08:43:50.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:49 smithi134 conmon[36363]: audit 2021-11-19T08:43:48.754393+0000 mon.smithi023 (mon.0) 768 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.opxllb"}]: dispatch 2021-11-19T08:43:50.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:49 smithi134 conmon[36363]: 2021-11-19T08:43:50.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:49 smithi134 conmon[36363]: cluster 2021-11-19T08:43: 2021-11-19T08:43:50.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:49 smithi134 conmon[36363]: 48.762899+0000 mon.smithi023 (mon.0) 769 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.fuvzkr=up:replay} 1 up:standby 2021-11-19T08:43:50.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:49 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:43:50.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:49 smithi134 conmon[36363]: 11-19T08:43:49.296472+0000 mon.smithi023 (mon.0) 770 : audit [DBG] from='client.? 172.21.15.23:0/1730209889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:50.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: cluster 2021-11-19T08:43:48.751679 2021-11-19T08:43:50.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 765 : cluster [DBG] mds.? [v2:172.21.15.134:6826/3725515320,v1:172.21.15.134:6827/3725515320] up:boot 2021-11-19T08:43:50.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: cluster 2021-11-19T08:43:48.751789+0000 mon.smithi023 (mon.0) 766 : 2021-11-19T08:43:50.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: cluster [INF] Standby daemon mds.cephfs.smithi023.fuvzkr assigned to filesystem cephfs as rank 1 2021-11-19T08:43:50.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: cluster 2021-11- 2021-11-19T08:43:50.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: 19T08:43:48.754168+0000 mon.smithi023 (mon.0) 767 : cluster 2021-11-19T08:43:50.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: [DBG] fsmap cephfs:1/2 {0=cephfs.smithi134.jqsnzo=up:resolve} 2 up:standby, 1 failed 2021-11-19T08:43:50.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: audit 2021-11-19T08:43:48. 2021-11-19T08:43:50.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: 754393+0000 mon.smithi023 (mon.0) 768 : audit 2021-11-19T08:43:50.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.opxllb"}]: dispatch 2021-11-19T08:43:50.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: cluster 2021-11-19T08:43:48.762899+0000 2021-11-19T08:43:50.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: mon.smithi023 (mon.0) 769 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.fuvzkr=up:replay} 1 up:standby 2021-11-19T08:43:50.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: 2021-11-19T08:43:50.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: audit 2021-11-19T08:43:49. 2021-11-19T08:43:50.146 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: 296472+0000 mon.smithi023 (mon.0) 770 : audit [DBG] 2021-11-19T08:43:50.147 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:49 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/1730209889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:50.295 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:50.430 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.441+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2693872382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cd40f5500 msgr2=0x7f3cd40f5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:50.430 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.441+0000 7f3cd3fff700 1 --2- 172.21.15.23:0/2693872382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cd40f5500 0x7f3cd40f5920 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f3ccc004660 tx=0x7f3ccc009b30).stop 2021-11-19T08:43:50.430 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.442+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2693872382 shutdown_connections 2021-11-19T08:43:50.432 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.442+0000 7f3cd3fff700 1 --2- 172.21.15.23:0/2693872382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cd40f67a0 0x7f3cd40f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:50.432 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.442+0000 7f3cd3fff700 1 --2- 172.21.15.23:0/2693872382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cd40f5500 0x7f3cd40f5920 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:50.432 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.442+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2693872382 >> 172.21.15.23:0/2693872382 conn(0x7f3cd40f0c40 msgr2=0x7f3cd40f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:50.432 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.442+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2693872382 shutdown_connections 2021-11-19T08:43:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.442+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2693872382 wait complete. 2021-11-19T08:43:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.442+0000 7f3cd3fff700 1 Processor -- start 2021-11-19T08:43:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.442+0000 7f3cd3fff700 1 -- start start 2021-11-19T08:43:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3cd3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cd40f5500 0x7f3cd4109a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:50.434 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3cd3fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cd40f67a0 0x7f3cd410bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:50.434 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3cd3fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd4060540 con 0x7f3cd40f5500 2021-11-19T08:43:50.434 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3cd3fff700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd4060680 con 0x7f3cd40f67a0 2021-11-19T08:43:50.434 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3cd2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cd40f5500 0x7f3cd4109a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:50.435 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3cd2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cd40f5500 0x7f3cd4109a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48102/0 (socket says 172.21.15.23:48102) 2021-11-19T08:43:50.435 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3cd2ffd700 1 -- 172.21.15.23:0/2332484567 learned_addr learned my addr 172.21.15.23:0/2332484567 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:50.435 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3cd27fc700 1 --2- 172.21.15.23:0/2332484567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cd40f67a0 0x7f3cd410bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:50.435 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3cd2ffd700 1 -- 172.21.15.23:0/2332484567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cd40f67a0 msgr2=0x7f3cd410bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3cd2ffd700 1 --2- 172.21.15.23:0/2332484567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cd40f67a0 0x7f3cd410bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3cd2ffd700 1 -- 172.21.15.23:0/2332484567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc005040 con 0x7f3cd40f5500 2021-11-19T08:43:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3cd2ffd700 1 --2- 172.21.15.23:0/2332484567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cd40f5500 0x7f3cd4109a80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f3ccc009f50 tx=0x7f3ccc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3ccb7fe700 1 -- 172.21.15.23:0/2332484567 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ccc018070 con 0x7f3cd40f5500 2021-11-19T08:43:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2332484567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd410c4d0 con 0x7f3cd40f5500 2021-11-19T08:43:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3ccb7fe700 1 -- 172.21.15.23:0/2332484567 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ccc013410 con 0x7f3cd40f5500 2021-11-19T08:43:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3ccb7fe700 1 -- 172.21.15.23:0/2332484567 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ccc01c620 con 0x7f3cd40f5500 2021-11-19T08:43:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.443+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2332484567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd410ca00 con 0x7f3cd40f5500 2021-11-19T08:43:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.444+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2332484567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd40f48f0 con 0x7f3cd40f5500 2021-11-19T08:43:50.438 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.445+0000 7f3ccb7fe700 1 -- 172.21.15.23:0/2332484567 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3ccc01d420 con 0x7f3cd40f5500 2021-11-19T08:43:50.439 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.445+0000 7f3ccb7fe700 1 --2- 172.21.15.23:0/2332484567 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3cc005c300 0x7f3cc005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:50.439 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.445+0000 7f3ccb7fe700 1 -- 172.21.15.23:0/2332484567 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f3ccc078160 con 0x7f3cd40f5500 2021-11-19T08:43:50.439 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.445+0000 7f3cd27fc700 1 --2- 172.21.15.23:0/2332484567 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3cc005c300 0x7f3cc005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:50.439 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.446+0000 7f3cd27fc700 1 --2- 172.21.15.23:0/2332484567 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3cc005c300 0x7f3cc005e7c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f3cbc007fd0 tx=0x7f3cbc009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:50.440 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.449+0000 7f3ccb7fe700 1 -- 172.21.15.23:0/2332484567 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ccc04aa20 con 0x7f3cd40f5500 2021-11-19T08:43:50.631 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.645+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2332484567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cd4045ad0 con 0x7f3cd40f5500 2021-11-19T08:43:50.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.646+0000 7f3ccb7fe700 1 -- 172.21.15.23:0/2332484567 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+2955 (secure 0 0 0) 0x7f3ccc04e040 con 0x7f3cd40f5500 2021-11-19T08:43:50.634 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:50.634 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":70},{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":70,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:49.759429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:resolve","state_seq":5,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:resolve","state_seq":10,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:50.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.647+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2332484567 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3cc005c300 msgr2=0x7f3cc005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:50.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.647+0000 7f3cd3fff700 1 --2- 172.21.15.23:0/2332484567 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3cc005c300 0x7f3cc005e7c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f3cbc007fd0 tx=0x7f3cbc009040).stop 2021-11-19T08:43:50.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.647+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2332484567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cd40f5500 msgr2=0x7f3cd4109a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:50.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.647+0000 7f3cd3fff700 1 --2- 172.21.15.23:0/2332484567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cd40f5500 0x7f3cd4109a80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f3ccc009f50 tx=0x7f3ccc009b30).stop 2021-11-19T08:43:50.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.647+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2332484567 shutdown_connections 2021-11-19T08:43:50.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.647+0000 7f3cd3fff700 1 --2- 172.21.15.23:0/2332484567 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3cc005c300 0x7f3cc005e7c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:50.639 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.647+0000 7f3cd3fff700 1 --2- 172.21.15.23:0/2332484567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cd40f5500 0x7f3cd4109a80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:50.639 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.647+0000 7f3cd3fff700 1 --2- 172.21.15.23:0/2332484567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3cd40f67a0 0x7f3cd410bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:50.640 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.647+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2332484567 >> 172.21.15.23:0/2332484567 conn(0x7f3cd40f0c40 msgr2=0x7f3cd40fa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:50.641 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.647+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2332484567 shutdown_connections 2021-11-19T08:43:50.641 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:50.647+0000 7f3cd3fff700 1 -- 172.21.15.23:0/2332484567 wait complete. 2021-11-19T08:43:50.641 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 70 2021-11-19T08:43:50.645 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:49.759429+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:resolve', 'state_seq': 5, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:resolve', 'state_seq': 10, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:50.645 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14732:up:resolve 2021-11-19T08:43:51.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:50 smithi134 conmon[36363]: cluster 2021-11-19T08:43:48.879930+0000 mgr.smithi023.ehlpcl (mgr.14164) 294 : cluster [DBG] pgmap v267: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:43:51.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:50 smithi134 conmon[36363]: cluster 2021-11-19T08:43:49.772389+0000 mon.smithi023 (mon.0) 771 : cluster [DBG] mds.? [v2:172.21.15.23:6826/4052910066,v1:172.21.15.23:6827/4052910066] up:resolve 2021-11-19T08:43:51.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:50 smithi134 conmon[36363]: cluster 2021-11-19T08:43:49. 2021-11-19T08:43:51.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:50 smithi134 conmon[36363]: 772441+0000 mon.smithi023 (mon.0) 772 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3188238627,v1:172.21.15.23:6829/3188238627] up:boot 2021-11-19T08:43:51.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:50 smithi134 conmon[36363]: cluster 2021-11-19T08: 2021-11-19T08:43:51.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:50 smithi134 conmon[36363]: 43:49.772500+0000 mon.smithi023 (mon.0) 2021-11-19T08:43:51.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:50 smithi134 conmon[36363]: 773 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.fuvzkr=up:resolve} 2 up:standby 2021-11-19T08:43:51.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:50 smithi134 conmon[36363]: audit 2021-11-19 2021-11-19T08:43:51.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:50 smithi134 conmon[36363]: T08:43:49.772785+0000 mon.smithi023 ( 2021-11-19T08:43:51.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:50 smithi134 conmon[36363]: mon.0) 774 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.wazanl"}]: dispatch 2021-11-19T08:43:51.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:50 smithi134 conmon[36363]: audit 2021-11 2021-11-19T08:43:51.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:50 smithi134 conmon[36363]: -19T08:43:50.647797+0000 mon.smithi023 2021-11-19T08:43:51.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:50 smithi134 conmon[36363]: (mon.0) 775 : audit [DBG] from='client.? 172.21.15.23:0/2332484567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:51.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: cluster 2021-11-19T08:43:48.879930+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:43:51.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: .14164) 294 : cluster [DBG] pgmap v267: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:43:51.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: cluster 2021-11-19T08:43:49.772389+0000 mon.smithi023 (mon 2021-11-19T08:43:51.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: .0) 771 : cluster [DBG] mds.? [v2:172.21.15.23:6826/4052910066,v1:172.21.15.23:6827/4052910066] up:resolve 2021-11-19T08:43:51.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: cluster 2021-11-19T08:43:49 2021-11-19T08:43:51.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: .772441+0000 mon.smithi023 (mon.0) 772 : cluster 2021-11-19T08:43:51.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: [DBG] mds.? [v2:172.21.15.23:6828/3188238627,v1:172.21.15.23:6829/3188238627] up:boot 2021-11-19T08:43:51.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: cluster 2021-11-19T08 2021-11-19T08:43:51.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: :43:49.772500+0000 mon.smithi023 (mon.0) 2021-11-19T08:43:51.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: 773 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.fuvzkr=up:resolve} 2 up:standby 2021-11-19T08:43:51.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:43:51.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: -19T08:43:49.772785+0000 mon.smithi023 ( 2021-11-19T08:43:51.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: mon.0) 774 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.wazanl"}]: dispatch 2021-11-19T08:43:51.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: audit 2021-11-19T08:43:51.145 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: 2021-11-19T08:43:50.647797+0000 2021-11-19T08:43:51.145 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: mon.smithi023 (mon.0) 775 : audit [DBG] 2021-11-19T08:43:51.145 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:50 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/2332484567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:51.647 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:52.003 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.016+0000 7f3f34c3b700 1 -- 172.21.15.23:0/3883781840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f300f4840 msgr2=0x7f3f300f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:52.004 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.016+0000 7f3f34c3b700 1 --2- 172.21.15.23:0/3883781840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f300f4840 0x7f3f300f4c60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f3f20004660 tx=0x7f3f20009b30).stop 2021-11-19T08:43:52.004 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.016+0000 7f3f34c3b700 1 -- 172.21.15.23:0/3883781840 shutdown_connections 2021-11-19T08:43:52.004 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.016+0000 7f3f34c3b700 1 --2- 172.21.15.23:0/3883781840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f300f51e0 0x7f3f300f7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:52.005 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.016+0000 7f3f34c3b700 1 --2- 172.21.15.23:0/3883781840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f300f4840 0x7f3f300f4c60 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:52.005 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.016+0000 7f3f34c3b700 1 -- 172.21.15.23:0/3883781840 >> 172.21.15.23:0/3883781840 conn(0x7f3f300f0360 msgr2=0x7f3f300f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:52.005 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.016+0000 7f3f34c3b700 1 -- 172.21.15.23:0/3883781840 shutdown_connections 2021-11-19T08:43:52.005 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.016+0000 7f3f34c3b700 1 -- 172.21.15.23:0/3883781840 wait complete. 2021-11-19T08:43:52.006 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.016+0000 7f3f34c3b700 1 Processor -- start 2021-11-19T08:43:52.006 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.016+0000 7f3f34c3b700 1 -- start start 2021-11-19T08:43:52.006 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f34c3b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f300f4840 0x7f3f30106e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:52.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f34c3b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f300f51e0 0x7f3f301072f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:52.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f34c3b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f301078f0 con 0x7f3f300f51e0 2021-11-19T08:43:52.007 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f34c3b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f30108100 con 0x7f3f300f4840 2021-11-19T08:43:52.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f2effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f300f51e0 0x7f3f301072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:52.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f2effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f300f51e0 0x7f3f301072f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48118/0 (socket says 172.21.15.23:48118) 2021-11-19T08:43:52.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f2effd700 1 -- 172.21.15.23:0/2704072189 learned_addr learned my addr 172.21.15.23:0/2704072189 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:52.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f2f7fe700 1 --2- 172.21.15.23:0/2704072189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f300f4840 0x7f3f30106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:52.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f2effd700 1 -- 172.21.15.23:0/2704072189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f300f4840 msgr2=0x7f3f30106e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:52.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f2effd700 1 --2- 172.21.15.23:0/2704072189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f300f4840 0x7f3f30106e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:52.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f2effd700 1 -- 172.21.15.23:0/2704072189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f20005040 con 0x7f3f300f51e0 2021-11-19T08:43:52.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f2effd700 1 --2- 172.21.15.23:0/2704072189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f300f51e0 0x7f3f301072f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f3f24002700 tx=0x7f3f24009d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:52.010 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f17fff700 1 -- 172.21.15.23:0/2704072189 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f240079c0 con 0x7f3f300f51e0 2021-11-19T08:43:52.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f17fff700 1 -- 172.21.15.23:0/2704072189 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f2400f040 con 0x7f3f300f51e0 2021-11-19T08:43:52.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f34c3b700 1 -- 172.21.15.23:0/2704072189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f30108380 con 0x7f3f300f51e0 2021-11-19T08:43:52.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f17fff700 1 -- 172.21.15.23:0/2704072189 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f24014740 con 0x7f3f300f51e0 2021-11-19T08:43:52.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.017+0000 7f3f34c3b700 1 -- 172.21.15.23:0/2704072189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f30108940 con 0x7f3f300f51e0 2021-11-19T08:43:52.012 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.018+0000 7f3f34c3b700 1 -- 172.21.15.23:0/2704072189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f300f3ff0 con 0x7f3f300f51e0 2021-11-19T08:43:52.013 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.019+0000 7f3f17fff700 1 -- 172.21.15.23:0/2704072189 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3f2401b060 con 0x7f3f300f51e0 2021-11-19T08:43:52.013 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.020+0000 7f3f17fff700 1 --2- 172.21.15.23:0/2704072189 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3f1805c300 0x7f3f1805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:52.013 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.020+0000 7f3f17fff700 1 -- 172.21.15.23:0/2704072189 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f3f24076340 con 0x7f3f300f51e0 2021-11-19T08:43:52.014 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.020+0000 7f3f2f7fe700 1 --2- 172.21.15.23:0/2704072189 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3f1805c300 0x7f3f1805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:52.014 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.020+0000 7f3f2f7fe700 1 --2- 172.21.15.23:0/2704072189 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3f1805c300 0x7f3f1805e7c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f3f30106b70 tx=0x7f3f20009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:52.014 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.023+0000 7f3f17fff700 1 -- 172.21.15.23:0/2704072189 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f24048c90 con 0x7f3f300f51e0 2021-11-19T08:43:52.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:51 smithi134 conmon[36363]: cluster 2021-11-19T08:43:50.786620+0000 mon.smithi023 (mon.0) 776 : cluster [DBG] mds.? [v2:172.21.15.23:6826/4052910066,v1:172.21.15.23:6827/4052910066] up:reconnect 2021-11-19T08:43:52.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:51 smithi134 conmon[36363]: cluster 2021-11-19T08:43:50.786656+0000 mon.smithi023 (mon.0) 777 : cluster [DBG] mds.? [v2:172.21.15.134:6824/464801254,v1:172.21.15.134:6825/464801254] up:reconnect 2021-11-19T08:43:52.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:51 smithi134 conmon[36363]: cluster 2021-11-19T08:43:50.786703+0000 mon.smithi023 (mon.0) 778 : cluster [DBG] 2021-11-19T08:43:52.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:51 smithi134 conmon[36363]: fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:reconnect,1=cephfs.smithi023.fuvzkr=up:reconnect} 2 up:standby 2021-11-19T08:43:52.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:51 smithi134 conmon[36363]: cluster 2021-11-19T08 2021-11-19T08:43:52.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:51 smithi134 conmon[36363]: :43:50.810429+0000 mds.cephfs.smithi023.fuvzkr (mds.1) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249997 2021-11-19T08:43:52.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:51 smithi134 conmon[36363]: cluster 2021-11-19T08:43:52.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:51 smithi134 conmon[36363]: 2021-11-19T08:43:50.811119 2021-11-19T08:43:52.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:51 smithi134 conmon[36363]: +0000 mds.cephfs.smithi134.jqsnzo (mds.0) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249994 2021-11-19T08:43:52.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:51 smithi134 conmon[36363]: 2021-11-19T08:43:52.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:51 smithi023 conmon[33490]: cluster 2021-11-19T08:43:50.786620+0000 mon.smithi023 (mon.0 2021-11-19T08:43:52.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:51 smithi023 conmon[33490]: ) 776 : cluster [DBG] mds.? [v2:172.21.15.23:6826/4052910066,v1:172.21.15.23:6827/4052910066] up:reconnect 2021-11-19T08:43:52.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:51 smithi023 conmon[33490]: cluster 2021-11-19T08:43:50.786656+0000 mon.smithi023 (mon.0) 777 : cluster [DBG] mds.? [v2:172.21.15.134:6824/464801254,v1:172.21.15.134:6825/464801254] up:reconnect 2021-11-19T08:43:52.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:51 smithi023 conmon[33490]: cluster 2021-11-19T08:43:50.786703+0000 mon.smithi023 (mon.0) 778 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:reconnect,1=cephfs.smithi023.fuvzkr=up:reconnect} 2 up:standby 2021-11-19T08:43:52.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:51 smithi023 conmon[33490]: cluster 2021-11 2021-11-19T08:43:52.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:51 smithi023 conmon[33490]: -19T08:43:50.810429+0000 mds.cephfs.smithi023.fuvzkr (mds.1) 1 : cluster [DBG] 2021-11-19T08:43:52.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:51 smithi023 conmon[33490]: reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249997 2021-11-19T08:43:52.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:51 smithi023 conmon[33490]: cluster 2021-11-19T08:43:52.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:51 smithi023 conmon[33490]: 2021-11-19T08: 2021-11-19T08:43:52.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:51 smithi023 conmon[33490]: 43:50.811119+0000 mds.cephfs.smithi134.jqsnzo (mds.0) 1 2021-11-19T08:43:52.144 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:51 smithi023 conmon[33490]: : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249994 2021-11-19T08:43:52.205 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.220+0000 7f3f34c3b700 1 -- 172.21.15.23:0/2704072189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f30045ad0 con 0x7f3f300f51e0 2021-11-19T08:43:52.206 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.220+0000 7f3f17fff700 1 -- 172.21.15.23:0/2704072189 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2953 (secure 0 0 0) 0x7f3f240112f0 con 0x7f3f300f51e0 2021-11-19T08:43:52.207 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:52.207 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":70},{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":68}],"filesystems":[{"mdsmap":{"epoch":72,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:51.785556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:52.208 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.222+0000 7f3f34c3b700 1 -- 172.21.15.23:0/2704072189 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3f1805c300 msgr2=0x7f3f1805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:52.209 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.222+0000 7f3f34c3b700 1 --2- 172.21.15.23:0/2704072189 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3f1805c300 0x7f3f1805e7c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f3f30106b70 tx=0x7f3f20009b30).stop 2021-11-19T08:43:52.209 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.222+0000 7f3f34c3b700 1 -- 172.21.15.23:0/2704072189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f300f51e0 msgr2=0x7f3f301072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:52.209 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.222+0000 7f3f34c3b700 1 --2- 172.21.15.23:0/2704072189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f300f51e0 0x7f3f301072f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f3f24002700 tx=0x7f3f24009d60).stop 2021-11-19T08:43:52.209 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.222+0000 7f3f34c3b700 1 -- 172.21.15.23:0/2704072189 shutdown_connections 2021-11-19T08:43:52.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.222+0000 7f3f34c3b700 1 --2- 172.21.15.23:0/2704072189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f300f4840 0x7f3f30106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:52.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.222+0000 7f3f34c3b700 1 --2- 172.21.15.23:0/2704072189 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3f1805c300 0x7f3f1805e7c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:52.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.222+0000 7f3f34c3b700 1 --2- 172.21.15.23:0/2704072189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f300f51e0 0x7f3f301072f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:52.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.222+0000 7f3f34c3b700 1 -- 172.21.15.23:0/2704072189 >> 172.21.15.23:0/2704072189 conn(0x7f3f300f0360 msgr2=0x7f3f300fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:52.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.222+0000 7f3f34c3b700 1 -- 172.21.15.23:0/2704072189 shutdown_connections 2021-11-19T08:43:52.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:52.222+0000 7f3f34c3b700 1 -- 172.21.15.23:0/2704072189 wait complete. 2021-11-19T08:43:52.211 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 72 2021-11-19T08:43:52.219 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:51.785556+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:rejoin', 'state_seq': 16, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:52.220 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14732:up:rejoin 2021-11-19T08:43:53.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:52 smithi134 conmon[36363]: cluster 2021-11-19T08:43:50.880528+0000 mgr.smithi023.ehlpcl (mgr.14164) 295 : cluster 2021-11-19T08:43:53.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:52 smithi134 conmon[36363]: [DBG] pgmap v268: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-19T08:43:53.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:52 smithi134 conmon[36363]: cluster 2021-11-19T08:43:51.802842+0000 mon.smithi023 (mon.0) 779 : cluster [DBG] mds.? [v2:172.21.15.23:6826/4052910066,v1:172.21.15.23:6827/4052910066] up:rejoin 2021-11-19T08:43:53.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:52 smithi134 conmon[36363]: cluster 2021-11-19T08:43: 2021-11-19T08:43:53.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:52 smithi134 conmon[36363]: 51.802914+0000 mon.smithi023 (mon.0) 780 : cluster [DBG] mds.? [v2:172.21.15.134:6824/464801254,v1:172.21.15.134:6825/464801254] up:rejoin 2021-11-19T08:43:53.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:52 smithi134 conmon[36363]: cluster 2021-11-19T08:43:51 2021-11-19T08:43:53.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:52 smithi134 conmon[36363]: .802996+0000 mon.smithi023 (mon.0) 781 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:rejoin,1=cephfs.smithi023.fuvzkr=up:rejoin} 2 up:standby 2021-11-19T08:43:53.041 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:52 smithi134 conmon[36363]: cluster 2021-11-19T08:43:53.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:52 smithi134 conmon[36363]: 2021-11-19T08:43:51.804452+0000 mon.smithi023 (mon.0) 782 : cluster [INF] daemon mds.cephfs.smithi023.fuvzkr is now active in filesystem cephfs as rank 1 2021-11-19T08:43:53.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:52 smithi134 conmon[36363]: 2021-11-19T08:43:53.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:52 smithi134 conmon[36363]: audit 2021-11-19 2021-11-19T08:43:53.042 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:52 smithi134 conmon[36363]: T08:43:52.222552+0000 mon.smithi023 (mon.0) 783 : audit [DBG] from='client.? 172.21.15.23:0/2704072189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:53.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:52 smithi023 conmon[33490]: cluster 2021-11-19T08:43:50.880528+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:43:53.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:52 smithi023 conmon[33490]: 14164) 295 : cluster [DBG] pgmap v268: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 818 B/s rd, 0 op/s 2021-11-19T08:43:53.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:52 smithi023 conmon[33490]: cluster 2021-11-19T08:43:51.802842+0000 mon.smithi023 (mon.0) 779 2021-11-19T08:43:53.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:52 smithi023 conmon[33490]: : cluster [DBG] mds.? [v2:172.21.15.23:6826/4052910066,v1:172.21.15.23:6827/4052910066] up:rejoin 2021-11-19T08:43:53.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:52 smithi023 conmon[33490]: cluster 2021-11-19T08:43:51.802914+0000 mon.smithi023 (mon.0) 780 : cluster [DBG] mds.? [v2:172.21.15.134:6824/464801254,v1:172.21.15.134:6825/464801254] up:rejoin 2021-11-19T08:43:53.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:52 smithi023 conmon[33490]: cluster 2021-11-19T08:43:51 2021-11-19T08:43:53.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:52 smithi023 conmon[33490]: .802996+0000 mon.smithi023 (mon.0) 781 : 2021-11-19T08:43:53.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:52 smithi023 conmon[33490]: cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:rejoin,1=cephfs.smithi023.fuvzkr=up:rejoin} 2 up:standby 2021-11-19T08:43:53.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:52 smithi023 conmon[33490]: cluster 2021-11-19T08:43:51.804452+0000 mon.smithi023 (mon.0) 782 : cluster [INF] daemon mds.cephfs.smithi023.fuvzkr is now active in filesystem cephfs as rank 1 2021-11-19T08:43:53.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:52 smithi023 conmon[33490]: audit 2021-11-19T08:43:52.222552+0000 mon.smithi023 (mon 2021-11-19T08:43:53.143 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:52 smithi023 conmon[33490]: .0) 783 : audit [DBG] from='client.? 172.21.15.23:0/2704072189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:53.221 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:53.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.376+0000 7f63afa7a700 1 -- 172.21.15.23:0/123554614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63a80f6680 msgr2=0x7f63a80f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:53.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.376+0000 7f63afa7a700 1 --2- 172.21.15.23:0/123554614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63a80f6680 0x7f63a80f6aa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f63a4004660 tx=0x7f63a4009b30).stop 2021-11-19T08:43:53.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.376+0000 7f63afa7a700 1 -- 172.21.15.23:0/123554614 shutdown_connections 2021-11-19T08:43:53.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.376+0000 7f63afa7a700 1 --2- 172.21.15.23:0/123554614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63a80f7920 0x7f63a80f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:53.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.376+0000 7f63afa7a700 1 --2- 172.21.15.23:0/123554614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63a80f6680 0x7f63a80f6aa0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:53.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.376+0000 7f63afa7a700 1 -- 172.21.15.23:0/123554614 >> 172.21.15.23:0/123554614 conn(0x7f63a80f1de0 msgr2=0x7f63a80f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:53.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.376+0000 7f63afa7a700 1 -- 172.21.15.23:0/123554614 shutdown_connections 2021-11-19T08:43:53.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.376+0000 7f63afa7a700 1 -- 172.21.15.23:0/123554614 wait complete. 2021-11-19T08:43:53.365 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f63afa7a700 1 Processor -- start 2021-11-19T08:43:53.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f63afa7a700 1 -- start start 2021-11-19T08:43:53.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f63afa7a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63a80f6680 0x7f63a8106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:53.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f63afa7a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63a80f7920 0x7f63a81073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:53.366 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f63afa7a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a8107950 con 0x7f63a80f6680 2021-11-19T08:43:53.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f63afa7a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a8107a90 con 0x7f63a80f7920 2021-11-19T08:43:53.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f63ad816700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63a80f6680 0x7f63a8106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:53.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f63ad816700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63a80f6680 0x7f63a8106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48130/0 (socket says 172.21.15.23:48130) 2021-11-19T08:43:53.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f63ad816700 1 -- 172.21.15.23:0/208915321 learned_addr learned my addr 172.21.15.23:0/208915321 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:53.367 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f63ad816700 1 -- 172.21.15.23:0/208915321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63a80f7920 msgr2=0x7f63a81073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:53.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f63ad816700 1 --2- 172.21.15.23:0/208915321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63a80f7920 0x7f63a81073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:43:53.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f63ad816700 1 -- 172.21.15.23:0/208915321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a4005040 con 0x7f63a80f6680 2021-11-19T08:43:53.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f63ad816700 1 --2- 172.21.15.23:0/208915321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63a80f6680 0x7f63a8106eb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f63a4009f50 tx=0x7f63a4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:53.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f639e7fc700 1 -- 172.21.15.23:0/208915321 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63a4018070 con 0x7f63a80f6680 2021-11-19T08:43:53.368 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f639e7fc700 1 -- 172.21.15.23:0/208915321 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63a4013410 con 0x7f63a80f6680 2021-11-19T08:43:53.369 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f639e7fc700 1 -- 172.21.15.23:0/208915321 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63a401c620 con 0x7f63a80f6680 2021-11-19T08:43:53.369 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.377+0000 7f63afa7a700 1 -- 172.21.15.23:0/208915321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a810c4d0 con 0x7f63a80f6680 2021-11-19T08:43:53.369 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.378+0000 7f63afa7a700 1 -- 172.21.15.23:0/208915321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a810ca90 con 0x7f63a80f6680 2021-11-19T08:43:53.370 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.379+0000 7f639e7fc700 1 -- 172.21.15.23:0/208915321 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f63a401d420 con 0x7f63a80f6680 2021-11-19T08:43:53.371 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.379+0000 7f63afa7a700 1 -- 172.21.15.23:0/208915321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a8027a00 con 0x7f63a80f6680 2021-11-19T08:43:53.371 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.379+0000 7f639e7fc700 1 --2- 172.21.15.23:0/208915321 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f639405c2a0 0x7f639405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:53.371 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.379+0000 7f639e7fc700 1 -- 172.21.15.23:0/208915321 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f63a40780b0 con 0x7f63a80f6680 2021-11-19T08:43:53.371 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.379+0000 7f63ad015700 1 --2- 172.21.15.23:0/208915321 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f639405c2a0 0x7f639405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:53.372 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.380+0000 7f63ad015700 1 --2- 172.21.15.23:0/208915321 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f639405c2a0 0x7f639405e760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f63980047d0 tx=0x7f6398007a40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:53.372 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.383+0000 7f639e7fc700 1 -- 172.21.15.23:0/208915321 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63a404a9f0 con 0x7f63a80f6680 2021-11-19T08:43:53.575 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.590+0000 7f63afa7a700 1 -- 172.21.15.23:0/208915321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63a80fee10 con 0x7f63a80f6680 2021-11-19T08:43:53.576 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.591+0000 7f639e7fc700 1 -- 172.21.15.23:0/208915321 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+2958 (secure 0 0 0) 0x7f63a404e010 con 0x7f63a80f6680 2021-11-19T08:43:53.577 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:53.577 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":70},{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":73,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:52.800166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:53.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.592+0000 7f63afa7a700 1 -- 172.21.15.23:0/208915321 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f639405c2a0 msgr2=0x7f639405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:53.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.592+0000 7f63afa7a700 1 --2- 172.21.15.23:0/208915321 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f639405c2a0 0x7f639405e760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f63980047d0 tx=0x7f6398007a40).stop 2021-11-19T08:43:53.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.592+0000 7f63afa7a700 1 -- 172.21.15.23:0/208915321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63a80f6680 msgr2=0x7f63a8106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:53.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.592+0000 7f63afa7a700 1 --2- 172.21.15.23:0/208915321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63a80f6680 0x7f63a8106eb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f63a4009f50 tx=0x7f63a4009b30).stop 2021-11-19T08:43:53.580 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.592+0000 7f63afa7a700 1 -- 172.21.15.23:0/208915321 shutdown_connections 2021-11-19T08:43:53.580 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.592+0000 7f63afa7a700 1 --2- 172.21.15.23:0/208915321 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f639405c2a0 0x7f639405e760 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:53.580 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.592+0000 7f63afa7a700 1 --2- 172.21.15.23:0/208915321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63a80f6680 0x7f63a8106eb0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:53.580 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.592+0000 7f63afa7a700 1 --2- 172.21.15.23:0/208915321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63a80f7920 0x7f63a81073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:43:53.581 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.592+0000 7f63afa7a700 1 -- 172.21.15.23:0/208915321 >> 172.21.15.23:0/208915321 conn(0x7f63a80f1de0 msgr2=0x7f63a80fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:53.581 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.593+0000 7f63afa7a700 1 -- 172.21.15.23:0/208915321 shutdown_connections 2021-11-19T08:43:53.582 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:53.593+0000 7f63afa7a700 1 -- 172.21.15.23:0/208915321 wait complete. 2021-11-19T08:43:53.582 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 73 2021-11-19T08:43:53.590 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:52.800166+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:clientreplay', 'state_seq': 17, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:53.590 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:clientreplay 2021-11-19T08:43:53.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:53 smithi023 conmon[33490]: cluster 2021-11-19T08:43:52.808287+0000 2021-11-19T08:43:53.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:53 smithi023 conmon[33490]: mon.smithi023 (mon.0) 784 : cluster [DBG] mds.? [v2:172.21.15.23:6826/4052910066,v1:172.21.15.23:6827/4052910066] up:active 2021-11-19T08:43:53.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:53 smithi023 conmon[33490]: cluster 2021-11-19T08:43:52.808345+0000 mon.smithi023 (mon.0) 785 : cluster [DBG] 2021-11-19T08:43:53.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:53 smithi023 conmon[33490]: mds.? [v2:172.21.15.134:6824/464801254,v1:172.21.15.134:6825/464801254] up:clientreplay 2021-11-19T08:43:53.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:53 smithi023 conmon[33490]: cluster 2021-11-19T08:43:52.808399+0000 mon.smithi023 (mon.0) 786 : cluster [DBG] mds.? [v2:172.21.15.134:6826/3725515320,v1:172.21.15.134:6827/3725515320] up:standby 2021-11-19T08:43:53.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:53 smithi023 conmon[33490]: 2021-11-19T08:43:53.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:53 smithi023 conmon[33490]: cluster 2021-11-19T08:43: 2021-11-19T08:43:53.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:53 smithi023 conmon[33490]: 52.808467+0000 mon.smithi023 (mon.0) 787 : cluster [DBG] 2021-11-19T08:43:53.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:53 smithi023 conmon[33490]: fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:clientreplay,1=cephfs.smithi023.fuvzkr=up:active} 2 up:standby 2021-11-19T08:43:53.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:53 smithi023 conmon[33490]: audit 2021 2021-11-19T08:43:53.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:53 smithi023 conmon[33490]: -11-19T08:43:53.592925+0000 mon.smithi023 (mon 2021-11-19T08:43:53.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:53 smithi023 conmon[33490]: .0) 788 : audit [DBG] 2021-11-19T08:43:53.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:53 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/208915321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:54.289 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:53 smithi134 conmon[36363]: cluster 2021-11-19T08:43:52.808287+0000 mon.smithi023 (mon.0) 784 : cluster [DBG] mds.? [v2:172.21.15.23:6826/4052910066,v1:172.21.15.23:6827/4052910066] up:active 2021-11-19T08:43:54.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:53 smithi134 conmon[36363]: cluster 2021-11-19T08:43:52.808345+0000 mon.smithi023 (mon.0) 785 : cluster [DBG] mds.? [v2:172.21.15.134:6824/464801254,v1:172.21.15.134:6825/464801254] up:clientreplay 2021-11-19T08:43:54.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:53 smithi134 conmon[36363]: cluster 2021-11-19T08:43:52.808399+0000 mon.smithi023 (mon.0) 786 : cluster [DBG] mds.? [v2:172.21.15.134:6826/3725515320,v1:172.21.15.134:6827/3725515320] up:standby 2021-11-19T08:43:54.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:53 smithi134 conmon[36363]: cluster 2021-11-19T08:43:52.808467+0000 mon.smithi023 (mon.0) 787 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:clientreplay,1=cephfs.smithi023.fuvzkr=up:active} 2 up:standby 2021-11-19T08:43:54.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:53 smithi134 conmon[36363]: audit 2021-11-19T08:43:53.592925+0000 mon.smithi023 (mon.0) 788 : audit [DBG] from='client.? 172.21.15.23:0/208915321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:54.592 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:54.727 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.738+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/2933048486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed00f6680 msgr2=0x7f0ed00f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:54.727 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.738+0000 7f0ed6b0d700 1 --2- 172.21.15.23:0/2933048486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed00f6680 0x7f0ed00f6aa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f0ec8004660 tx=0x7f0ec8009b30).stop 2021-11-19T08:43:54.728 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.738+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/2933048486 shutdown_connections 2021-11-19T08:43:54.728 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.738+0000 7f0ed6b0d700 1 --2- 172.21.15.23:0/2933048486 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ed00f7920 0x7f0ed00f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-11-19T08:43:54.728 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.738+0000 7f0ed6b0d700 1 --2- 172.21.15.23:0/2933048486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed00f6680 0x7f0ed00f6aa0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:54.728 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.738+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/2933048486 >> 172.21.15.23:0/2933048486 conn(0x7f0ed00f1de0 msgr2=0x7f0ed00f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:54.729 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.738+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/2933048486 shutdown_connections 2021-11-19T08:43:54.729 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.738+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/2933048486 wait complete. 2021-11-19T08:43:54.729 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.739+0000 7f0ed6b0d700 1 Processor -- start 2021-11-19T08:43:54.729 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.739+0000 7f0ed6b0d700 1 -- start start 2021-11-19T08:43:54.730 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.739+0000 7f0ed6b0d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ed00f6680 0x7f0ed0106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:54.730 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.739+0000 7f0ed6b0d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed00f7920 0x7f0ed01073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:54.730 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.739+0000 7f0ed6b0d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed0107a10 con 0x7f0ed00f7920 2021-11-19T08:43:54.731 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.739+0000 7f0ed6b0d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed0107b50 con 0x7f0ed00f6680 2021-11-19T08:43:54.731 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.739+0000 7f0ecffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed00f7920 0x7f0ed01073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:54.731 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.739+0000 7f0ecffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed00f7920 0x7f0ed01073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48144/0 (socket says 172.21.15.23:48144) 2021-11-19T08:43:54.731 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.739+0000 7f0ecffff700 1 -- 172.21.15.23:0/500024024 learned_addr learned my addr 172.21.15.23:0/500024024 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.739+0000 7f0ecffff700 1 -- 172.21.15.23:0/500024024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ed00f6680 msgr2=0x7f0ed0106ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.740+0000 7f0ecffff700 1 --2- 172.21.15.23:0/500024024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ed00f6680 0x7f0ed0106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:43:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.740+0000 7f0ecffff700 1 -- 172.21.15.23:0/500024024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec8005040 con 0x7f0ed00f7920 2021-11-19T08:43:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.740+0000 7f0ecffff700 1 --2- 172.21.15.23:0/500024024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed00f7920 0x7f0ed01073d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f0ec000b790 tx=0x7f0ec00092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.740+0000 7f0ecd7fa700 1 -- 172.21.15.23:0/500024024 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ec000d780 con 0x7f0ed00f7920 2021-11-19T08:43:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.740+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/500024024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed010c5f0 con 0x7f0ed00f7920 2021-11-19T08:43:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.740+0000 7f0ecd7fa700 1 -- 172.21.15.23:0/500024024 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ec000dd90 con 0x7f0ed00f7920 2021-11-19T08:43:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.740+0000 7f0ecd7fa700 1 -- 172.21.15.23:0/500024024 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ec000fea0 con 0x7f0ed00f7920 2021-11-19T08:43:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.740+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/500024024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed010cbb0 con 0x7f0ed00f7920 2021-11-19T08:43:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.741+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/500024024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed0027a00 con 0x7f0ed00f7920 2021-11-19T08:43:54.735 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.741+0000 7f0ecd7fa700 1 -- 172.21.15.23:0/500024024 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0ec000d8e0 con 0x7f0ed00f7920 2021-11-19T08:43:54.735 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.741+0000 7f0ecd7fa700 1 --2- 172.21.15.23:0/500024024 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0ebc05c2a0 0x7f0ebc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:54.736 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.742+0000 7f0ecd7fa700 1 -- 172.21.15.23:0/500024024 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f0ec0076c40 con 0x7f0ed00f7920 2021-11-19T08:43:54.736 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.742+0000 7f0ed48a9700 1 --2- 172.21.15.23:0/500024024 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0ebc05c2a0 0x7f0ebc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:54.737 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.742+0000 7f0ed48a9700 1 --2- 172.21.15.23:0/500024024 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0ebc05c2a0 0x7f0ebc05e760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f0ec8009f50 tx=0x7f0ec8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:54.737 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.746+0000 7f0ecd7fa700 1 -- 172.21.15.23:0/500024024 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ec0049500 con 0x7f0ed00f7920 2021-11-19T08:43:54.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:54 smithi023 conmon[33490]: cluster 2021-11-19T08:43:52.881025+0000 mgr.smithi023.ehlpcl 2021-11-19T08:43:54.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:54 smithi023 conmon[33490]: (mgr.14164) 296 : cluster [DBG] pgmap v269: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:43:54.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:54 smithi023 conmon[33490]: cluster 2021-11-19T08:43:53.831092+0000 2021-11-19T08:43:54.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:54 smithi023 conmon[33490]: mon.smithi023 (mon.0) 789 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3188238627,v1:172.21.15.23:6829/3188238627] up:standby 2021-11-19T08:43:54.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:54 smithi023 conmon[33490]: cluster 2021-11-19T08:43:54.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:54 smithi023 conmon[33490]: 2021-11-19 2021-11-19T08:43:54.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:54 smithi023 conmon[33490]: T08:43:53.831164 2021-11-19T08:43:54.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:54 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 790 : cluster 2021-11-19T08:43:54.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:54 smithi023 conmon[33490]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:clientreplay,1=cephfs.smithi023.fuvzkr=up:active} 2 up:standby 2021-11-19T08:43:54.929 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.943+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/500024024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ed00fee90 con 0x7f0ed00f7920 2021-11-19T08:43:54.930 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.944+0000 7f0ecd7fa700 1 -- 172.21.15.23:0/500024024 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+2957 (secure 0 0 0) 0x7f0ec004cb20 con 0x7f0ed00f7920 2021-11-19T08:43:54.930 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:54.930 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":74},{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":73,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:52.800166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:54.932 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.945+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/500024024 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0ebc05c2a0 msgr2=0x7f0ebc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:54.932 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.945+0000 7f0ed6b0d700 1 --2- 172.21.15.23:0/500024024 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0ebc05c2a0 0x7f0ebc05e760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f0ec8009f50 tx=0x7f0ec8009b30).stop 2021-11-19T08:43:54.932 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.946+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/500024024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed00f7920 msgr2=0x7f0ed01073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:54.932 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.946+0000 7f0ed6b0d700 1 --2- 172.21.15.23:0/500024024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed00f7920 0x7f0ed01073d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f0ec000b790 tx=0x7f0ec00092d0).stop 2021-11-19T08:43:54.933 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.946+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/500024024 shutdown_connections 2021-11-19T08:43:54.933 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.946+0000 7f0ed6b0d700 1 --2- 172.21.15.23:0/500024024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ed00f6680 0x7f0ed0106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:43:54.933 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.946+0000 7f0ed6b0d700 1 --2- 172.21.15.23:0/500024024 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0ebc05c2a0 0x7f0ebc05e760 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:54.933 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.946+0000 7f0ed6b0d700 1 --2- 172.21.15.23:0/500024024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed00f7920 0x7f0ed01073d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:54.934 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.946+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/500024024 >> 172.21.15.23:0/500024024 conn(0x7f0ed00f1de0 msgr2=0x7f0ed00fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:54.934 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.946+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/500024024 shutdown_connections 2021-11-19T08:43:54.934 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:54.946+0000 7f0ed6b0d700 1 -- 172.21.15.23:0/500024024 wait complete. 2021-11-19T08:43:54.935 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 74 2021-11-19T08:43:54.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:52.800166+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:clientreplay', 'state_seq': 17, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:54.943 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:clientreplay 2021-11-19T08:43:55.289 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:54 smithi134 conmon[36363]: cluster 2021-11-19T08:43:52.881025+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:43:55.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:54 smithi134 conmon[36363]: ) 296 : cluster [DBG] pgmap v269: 65 pgs: 65 active+clean; 4.9 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:43:55.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:54 smithi134 conmon[36363]: cluster 2021-11-19T08:43:53.831092+0000 mon.smithi023 (mon.0) 789 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3188238627,v1:172.21.15.23:6829/3188238627] up:standby 2021-11-19T08:43:55.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:54 smithi134 conmon[36363]: cluster 2021-11-19T08:43:53.831164+0000 mon.smithi023 (mon.0) 790 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:clientreplay,1=cephfs.smithi023.fuvzkr=up:active} 2 up:standby 2021-11-19T08:43:55.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:56.075 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.086+0000 7f090df97700 1 -- 172.21.15.23:0/3792260098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09080f4840 msgr2=0x7f09080f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:56.075 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.086+0000 7f090df97700 1 --2- 172.21.15.23:0/3792260098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09080f4840 0x7f09080f4c60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f08fc004660 tx=0x7f08fc009b30).stop 2021-11-19T08:43:56.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.086+0000 7f090df97700 1 -- 172.21.15.23:0/3792260098 shutdown_connections 2021-11-19T08:43:56.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.086+0000 7f090df97700 1 --2- 172.21.15.23:0/3792260098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09080f51e0 0x7f09080f7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:56.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.086+0000 7f090df97700 1 --2- 172.21.15.23:0/3792260098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09080f4840 0x7f09080f4c60 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:56.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.086+0000 7f090df97700 1 -- 172.21.15.23:0/3792260098 >> 172.21.15.23:0/3792260098 conn(0x7f09080f0360 msgr2=0x7f09080f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:56.076 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.087+0000 7f090df97700 1 -- 172.21.15.23:0/3792260098 shutdown_connections 2021-11-19T08:43:56.077 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.087+0000 7f090df97700 1 -- 172.21.15.23:0/3792260098 wait complete. 2021-11-19T08:43:56.077 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.087+0000 7f090df97700 1 Processor -- start 2021-11-19T08:43:56.077 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.087+0000 7f090df97700 1 -- start start 2021-11-19T08:43:56.077 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.087+0000 7f090df97700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09080f4840 0x7f0908106e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:56.078 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.087+0000 7f090df97700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09080f51e0 0x7f0908107310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:56.078 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.087+0000 7f090df97700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0908107910 con 0x7f09080f51e0 2021-11-19T08:43:56.078 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.087+0000 7f090df97700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0908108120 con 0x7f09080f4840 2021-11-19T08:43:56.078 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.088+0000 7f0907fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09080f51e0 0x7f0908107310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:56.079 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.088+0000 7f0907fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09080f51e0 0x7f0908107310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48158/0 (socket says 172.21.15.23:48158) 2021-11-19T08:43:56.079 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.088+0000 7f0907fff700 1 -- 172.21.15.23:0/776445053 learned_addr learned my addr 172.21.15.23:0/776445053 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:56.079 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.088+0000 7f090cf95700 1 --2- 172.21.15.23:0/776445053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09080f4840 0x7f0908106e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:56.079 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.088+0000 7f0907fff700 1 -- 172.21.15.23:0/776445053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09080f4840 msgr2=0x7f0908106e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:56.080 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.088+0000 7f0907fff700 1 --2- 172.21.15.23:0/776445053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09080f4840 0x7f0908106e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:56.080 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.088+0000 7f0907fff700 1 -- 172.21.15.23:0/776445053 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08fc005040 con 0x7f09080f51e0 2021-11-19T08:43:56.080 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.088+0000 7f0907fff700 1 --2- 172.21.15.23:0/776445053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09080f51e0 0x7f0908107310 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f08f4002700 tx=0x7f08f4009da0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:56.080 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.088+0000 7f09057fa700 1 -- 172.21.15.23:0/776445053 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08f40079c0 con 0x7f09080f51e0 2021-11-19T08:43:56.080 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.088+0000 7f09057fa700 1 -- 172.21.15.23:0/776445053 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08f400f040 con 0x7f09080f51e0 2021-11-19T08:43:56.081 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.088+0000 7f090df97700 1 -- 172.21.15.23:0/776445053 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09081083a0 con 0x7f09080f51e0 2021-11-19T08:43:56.081 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.088+0000 7f09057fa700 1 -- 172.21.15.23:0/776445053 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08f40147e0 con 0x7f09080f51e0 2021-11-19T08:43:56.081 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.088+0000 7f090df97700 1 -- 172.21.15.23:0/776445053 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0908108960 con 0x7f09080f51e0 2021-11-19T08:43:56.081 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.089+0000 7f090df97700 1 -- 172.21.15.23:0/776445053 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09080f3ff0 con 0x7f09080f51e0 2021-11-19T08:43:56.081 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.089+0000 7f09057fa700 1 -- 172.21.15.23:0/776445053 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f08f401b060 con 0x7f09080f51e0 2021-11-19T08:43:56.082 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.090+0000 7f09057fa700 1 --2- 172.21.15.23:0/776445053 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f08f805c300 0x7f08f805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:56.082 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.090+0000 7f09057fa700 1 -- 172.21.15.23:0/776445053 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f08f4076070 con 0x7f09080f51e0 2021-11-19T08:43:56.082 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.090+0000 7f090cf95700 1 --2- 172.21.15.23:0/776445053 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f08f805c300 0x7f08f805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:56.083 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.090+0000 7f090cf95700 1 --2- 172.21.15.23:0/776445053 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f08f805c300 0x7f08f805e7c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f0908106b90 tx=0x7f08fc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:56.083 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.094+0000 7f09057fa700 1 -- 172.21.15.23:0/776445053 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08f40489b0 con 0x7f09080f51e0 2021-11-19T08:43:56.112 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:55 smithi023 conmon[33490]: audit 2021-11-19T08:43:54.946090+0000 mon.smithi023 (mon.0) 791 : audit [DBG] 2021-11-19T08:43:56.112 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:55 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/500024024' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:56.267 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:55 smithi134 conmon[36363]: audit 2021-11-19T08:43:54.946090+0000 mon.smithi023 (mon.0) 791 : audit [DBG] from='client.? 172.21.15.23:0/500024024' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:56.272 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.287+0000 7f090df97700 1 -- 172.21.15.23:0/776445053 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0908045ad0 con 0x7f09080f51e0 2021-11-19T08:43:56.273 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.289+0000 7f09057fa700 1 -- 172.21.15.23:0/776445053 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+2957 (secure 0 0 0) 0x7f08f404bfd0 con 0x7f09080f51e0 2021-11-19T08:43:56.274 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:56.274 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":74},{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":73,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:52.800166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.289+0000 7f090df97700 1 -- 172.21.15.23:0/776445053 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f08f805c300 msgr2=0x7f08f805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.289+0000 7f090df97700 1 --2- 172.21.15.23:0/776445053 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f08f805c300 0x7f08f805e7c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f0908106b90 tx=0x7f08fc009b30).stop 2021-11-19T08:43:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.289+0000 7f090df97700 1 -- 172.21.15.23:0/776445053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09080f51e0 msgr2=0x7f0908107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.289+0000 7f090df97700 1 --2- 172.21.15.23:0/776445053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09080f51e0 0x7f0908107310 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f08f4002700 tx=0x7f08f4009da0).stop 2021-11-19T08:43:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.290+0000 7f090df97700 1 -- 172.21.15.23:0/776445053 shutdown_connections 2021-11-19T08:43:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.290+0000 7f090df97700 1 --2- 172.21.15.23:0/776445053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09080f4840 0x7f0908106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:56.277 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.290+0000 7f090df97700 1 --2- 172.21.15.23:0/776445053 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f08f805c300 0x7f08f805e7c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:56.277 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.290+0000 7f090df97700 1 --2- 172.21.15.23:0/776445053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09080f51e0 0x7f0908107310 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:56.277 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.290+0000 7f090df97700 1 -- 172.21.15.23:0/776445053 >> 172.21.15.23:0/776445053 conn(0x7f09080f0360 msgr2=0x7f09080fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:56.277 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.290+0000 7f090df97700 1 -- 172.21.15.23:0/776445053 shutdown_connections 2021-11-19T08:43:56.278 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:56.290+0000 7f090df97700 1 -- 172.21.15.23:0/776445053 wait complete. 2021-11-19T08:43:56.278 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 74 2021-11-19T08:43:56.287 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:52.800166+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:clientreplay', 'state_seq': 17, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:56.288 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:clientreplay 2021-11-19T08:43:57.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:56 smithi023 conmon[33490]: cluster 2021-11-19T08:43:54.881432+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:43:57.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:56 smithi023 conmon[33490]: mgr.14164) 297 : cluster [DBG] pgmap v270: 65 pgs: 65 active+clean; 5.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2021-11-19T08:43:57.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:56 smithi023 conmon[33490]: audit 2021-11-19T08:43:56.290014+0000 mon.smithi023 (mon.0) 792 : audit [DBG] from='client.? 172.21.15.23:0/776445053' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:57.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:57.293 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:56 smithi134 conmon[36363]: cluster 2021-11-19T08:43:54.881432+0000 mgr.smithi023.ehlpcl (mgr.14164) 297 2021-11-19T08:43:57.294 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:56 smithi134 conmon[36363]: : cluster [DBG] pgmap v270: 65 pgs: 65 active+clean; 5.0 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2021-11-19T08:43:57.295 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:56 smithi134 conmon[36363]: audit 2021-11-19T08:43:56.290014+0000 mon.smithi023 (mon.0) 792 : audit [DBG] from='client.? 172.21.15.23:0/776445053' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:57.428 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.442+0000 7f351a9bb700 1 -- 172.21.15.23:0/523328904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35140f6840 msgr2=0x7f35140f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:57.430 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.442+0000 7f351a9bb700 1 --2- 172.21.15.23:0/523328904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35140f6840 0x7f35140f6ca0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f3508004660 tx=0x7f3508009b30).stop 2021-11-19T08:43:57.430 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.442+0000 7f351a9bb700 1 -- 172.21.15.23:0/523328904 shutdown_connections 2021-11-19T08:43:57.431 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.442+0000 7f351a9bb700 1 --2- 172.21.15.23:0/523328904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35140f6840 0x7f35140f6ca0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:57.431 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.442+0000 7f351a9bb700 1 --2- 172.21.15.23:0/523328904 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35140f55a0 0x7f35140f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:57.431 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.442+0000 7f351a9bb700 1 -- 172.21.15.23:0/523328904 >> 172.21.15.23:0/523328904 conn(0x7f35140f0ce0 msgr2=0x7f35140f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:57.431 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.442+0000 7f351a9bb700 1 -- 172.21.15.23:0/523328904 shutdown_connections 2021-11-19T08:43:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.442+0000 7f351a9bb700 1 -- 172.21.15.23:0/523328904 wait complete. 2021-11-19T08:43:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.442+0000 7f351a9bb700 1 Processor -- start 2021-11-19T08:43:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.443+0000 7f351a9bb700 1 -- start start 2021-11-19T08:43:57.432 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.443+0000 7f351a9bb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35140f55a0 0x7f351405ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:57.433 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.443+0000 7f351a9bb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35140f6840 0x7f351410bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:57.433 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.443+0000 7f351a9bb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351410c180 con 0x7f35140f55a0 2021-11-19T08:43:57.433 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.443+0000 7f351a9bb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351410c2c0 con 0x7f35140f6840 2021-11-19T08:43:57.433 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.443+0000 7f3513fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35140f55a0 0x7f351405ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:57.434 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.443+0000 7f3513fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35140f55a0 0x7f351405ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48168/0 (socket says 172.21.15.23:48168) 2021-11-19T08:43:57.434 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.443+0000 7f3513fff700 1 -- 172.21.15.23:0/1164836163 learned_addr learned my addr 172.21.15.23:0/1164836163 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:57.434 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.443+0000 7f35137fe700 1 --2- 172.21.15.23:0/1164836163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35140f6840 0x7f351410bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:57.434 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.443+0000 7f3513fff700 1 -- 172.21.15.23:0/1164836163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35140f6840 msgr2=0x7f351410bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:57.434 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.443+0000 7f3513fff700 1 --2- 172.21.15.23:0/1164836163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35140f6840 0x7f351410bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:57.435 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.443+0000 7f3513fff700 1 -- 172.21.15.23:0/1164836163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3508005040 con 0x7f35140f55a0 2021-11-19T08:43:57.435 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.443+0000 7f3513fff700 1 --2- 172.21.15.23:0/1164836163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35140f55a0 0x7f351405ff20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f350400c560 tx=0x7f3504007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:57.435 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.443+0000 7f3510ff9700 1 -- 172.21.15.23:0/1164836163 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f350400f650 con 0x7f35140f55a0 2021-11-19T08:43:57.435 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.444+0000 7f351a9bb700 1 -- 172.21.15.23:0/1164836163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351410c520 con 0x7f35140f55a0 2021-11-19T08:43:57.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.444+0000 7f3510ff9700 1 -- 172.21.15.23:0/1164836163 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f350400fc60 con 0x7f35140f55a0 2021-11-19T08:43:57.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.444+0000 7f3510ff9700 1 -- 172.21.15.23:0/1164836163 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3504012d20 con 0x7f35140f55a0 2021-11-19T08:43:57.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.444+0000 7f351a9bb700 1 -- 172.21.15.23:0/1164836163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351410cae0 con 0x7f35140f55a0 2021-11-19T08:43:57.436 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.444+0000 7f351a9bb700 1 -- 172.21.15.23:0/1164836163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3514027a00 con 0x7f35140f55a0 2021-11-19T08:43:57.438 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.445+0000 7f3510ff9700 1 -- 172.21.15.23:0/1164836163 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3504019030 con 0x7f35140f55a0 2021-11-19T08:43:57.438 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.445+0000 7f3510ff9700 1 --2- 172.21.15.23:0/1164836163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f34fc05c2a0 0x7f34fc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:57.438 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.445+0000 7f3510ff9700 1 -- 172.21.15.23:0/1164836163 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f3504075f70 con 0x7f35140f55a0 2021-11-19T08:43:57.438 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.445+0000 7f35137fe700 1 --2- 172.21.15.23:0/1164836163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f34fc05c2a0 0x7f34fc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:57.439 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.446+0000 7f35137fe700 1 --2- 172.21.15.23:0/1164836163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f34fc05c2a0 0x7f34fc05e760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f3508009d60 tx=0x7f3508009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:57.439 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.449+0000 7f3510ff9700 1 -- 172.21.15.23:0/1164836163 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35040a22b0 con 0x7f35140f55a0 2021-11-19T08:43:57.631 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.646+0000 7f351a9bb700 1 -- 172.21.15.23:0/1164836163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35140fdd30 con 0x7f35140f55a0 2021-11-19T08:43:57.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.647+0000 7f3510ff9700 1 -- 172.21.15.23:0/1164836163 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+2957 (secure 0 0 0) 0x7f3504017070 con 0x7f35140f55a0 2021-11-19T08:43:57.632 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:57.632 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":74},{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":73,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:52.800166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:57.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.648+0000 7f351a9bb700 1 -- 172.21.15.23:0/1164836163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f34fc05c2a0 msgr2=0x7f34fc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:57.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.648+0000 7f351a9bb700 1 --2- 172.21.15.23:0/1164836163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f34fc05c2a0 0x7f34fc05e760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f3508009d60 tx=0x7f3508009b30).stop 2021-11-19T08:43:57.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.648+0000 7f351a9bb700 1 -- 172.21.15.23:0/1164836163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35140f55a0 msgr2=0x7f351405ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:57.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.648+0000 7f351a9bb700 1 --2- 172.21.15.23:0/1164836163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35140f55a0 0x7f351405ff20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f350400c560 tx=0x7f3504007e10).stop 2021-11-19T08:43:57.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.648+0000 7f351a9bb700 1 -- 172.21.15.23:0/1164836163 shutdown_connections 2021-11-19T08:43:57.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.648+0000 7f351a9bb700 1 --2- 172.21.15.23:0/1164836163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f34fc05c2a0 0x7f34fc05e760 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:57.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.648+0000 7f351a9bb700 1 --2- 172.21.15.23:0/1164836163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35140f55a0 0x7f351405ff20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:57.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.648+0000 7f351a9bb700 1 --2- 172.21.15.23:0/1164836163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35140f6840 0x7f351410bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:57.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.648+0000 7f351a9bb700 1 -- 172.21.15.23:0/1164836163 >> 172.21.15.23:0/1164836163 conn(0x7f35140f0ce0 msgr2=0x7f35140fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:57.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.648+0000 7f351a9bb700 1 -- 172.21.15.23:0/1164836163 shutdown_connections 2021-11-19T08:43:57.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:57.648+0000 7f351a9bb700 1 -- 172.21.15.23:0/1164836163 wait complete. 2021-11-19T08:43:57.637 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 74 2021-11-19T08:43:57.645 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:52.800166+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:clientreplay', 'state_seq': 17, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:57.646 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:clientreplay 2021-11-19T08:43:58.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:57 smithi023 conmon[33490]: cluster 2021-11-19T08:43:56.882038+0000 mgr.smithi023.ehlpcl (mgr.14164) 298 : cluster [DBG] pgmap v271: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-19T08:43:58.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:57 smithi023 conmon[33490]: audit 2021-11-19T08:43:57.647700+0000 mon.smithi023 (mon.0) 793 : audit [DBG] from='client.? 172.21.15.23:0/1164836163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:58.289 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:57 smithi134 conmon[36363]: cluster 2021-11-19T08:43:56.882038+0000 mgr.smithi023.ehlpcl (mgr.14164) 298 : cluster [DBG] pgmap v271: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-19T08:43:58.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:57 smithi134 conmon[36363]: audit 2021-11-19T08:43:57.647700+0000 mon.smithi023 (mon.0) 793 : audit [DBG] from='client.? 172.21.15.23:0/1164836163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:43:58.647 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:43:58.782 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.795+0000 7fa8daa62700 1 -- 172.21.15.23:0/2981528927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8d40f6770 msgr2=0x7fa8d40f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:58.784 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.795+0000 7fa8daa62700 1 --2- 172.21.15.23:0/2981528927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8d40f6770 0x7fa8d40f6bd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fa8c4004660 tx=0x7fa8c4009b30).stop 2021-11-19T08:43:58.784 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.796+0000 7fa8daa62700 1 -- 172.21.15.23:0/2981528927 shutdown_connections 2021-11-19T08:43:58.785 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.796+0000 7fa8daa62700 1 --2- 172.21.15.23:0/2981528927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8d40f6770 0x7fa8d40f6bd0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:58.785 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.796+0000 7fa8daa62700 1 --2- 172.21.15.23:0/2981528927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8d40f54d0 0x7fa8d40f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:58.785 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.796+0000 7fa8daa62700 1 -- 172.21.15.23:0/2981528927 >> 172.21.15.23:0/2981528927 conn(0x7fa8d40f0c30 msgr2=0x7fa8d40f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:58.785 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.796+0000 7fa8daa62700 1 -- 172.21.15.23:0/2981528927 shutdown_connections 2021-11-19T08:43:58.786 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.796+0000 7fa8daa62700 1 -- 172.21.15.23:0/2981528927 wait complete. 2021-11-19T08:43:58.786 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.796+0000 7fa8daa62700 1 Processor -- start 2021-11-19T08:43:58.786 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.796+0000 7fa8daa62700 1 -- start start 2021-11-19T08:43:58.786 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.796+0000 7fa8daa62700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8d40f54d0 0x7fa8d4105cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:58.787 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.796+0000 7fa8daa62700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8d40f6770 0x7fa8d41061b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:58.787 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.796+0000 7fa8daa62700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d4106760 con 0x7fa8d40f6770 2021-11-19T08:43:58.787 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.796+0000 7fa8daa62700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d4106f60 con 0x7fa8d40f54d0 2021-11-19T08:43:58.787 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.796+0000 7fa8d925f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8d40f6770 0x7fa8d41061b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:58.788 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.797+0000 7fa8d925f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8d40f6770 0x7fa8d41061b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48180/0 (socket says 172.21.15.23:48180) 2021-11-19T08:43:58.788 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.797+0000 7fa8d925f700 1 -- 172.21.15.23:0/1019634100 learned_addr learned my addr 172.21.15.23:0/1019634100 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:43:58.788 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.797+0000 7fa8d9a60700 1 --2- 172.21.15.23:0/1019634100 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8d40f54d0 0x7fa8d4105cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:58.788 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.797+0000 7fa8d925f700 1 -- 172.21.15.23:0/1019634100 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8d40f54d0 msgr2=0x7fa8d4105cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:58.788 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.797+0000 7fa8d925f700 1 --2- 172.21.15.23:0/1019634100 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8d40f54d0 0x7fa8d4105cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:58.789 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.797+0000 7fa8d925f700 1 -- 172.21.15.23:0/1019634100 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c4005040 con 0x7fa8d40f6770 2021-11-19T08:43:58.789 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.797+0000 7fa8d9a60700 1 --2- 172.21.15.23:0/1019634100 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8d40f54d0 0x7fa8d4105cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:43:58.789 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.797+0000 7fa8d925f700 1 --2- 172.21.15.23:0/1019634100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8d40f6770 0x7fa8d41061b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fa8c4009f50 tx=0x7fa8c4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:58.789 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.797+0000 7fa8ca7fc700 1 -- 172.21.15.23:0/1019634100 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8c4018070 con 0x7fa8d40f6770 2021-11-19T08:43:58.790 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.797+0000 7fa8ca7fc700 1 -- 172.21.15.23:0/1019634100 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8c4013410 con 0x7fa8d40f6770 2021-11-19T08:43:58.790 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.797+0000 7fa8daa62700 1 -- 172.21.15.23:0/1019634100 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d41071e0 con 0x7fa8d40f6770 2021-11-19T08:43:58.790 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.797+0000 7fa8ca7fc700 1 -- 172.21.15.23:0/1019634100 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8c401c3c0 con 0x7fa8d40f6770 2021-11-19T08:43:58.790 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.797+0000 7fa8daa62700 1 -- 172.21.15.23:0/1019634100 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d4107740 con 0x7fa8d40f6770 2021-11-19T08:43:58.791 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.798+0000 7fa8daa62700 1 -- 172.21.15.23:0/1019634100 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d40f48c0 con 0x7fa8d40f6770 2021-11-19T08:43:58.792 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.798+0000 7fa8ca7fc700 1 -- 172.21.15.23:0/1019634100 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa8c401d420 con 0x7fa8d40f6770 2021-11-19T08:43:58.792 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.799+0000 7fa8ca7fc700 1 --2- 172.21.15.23:0/1019634100 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa8c005c300 0x7fa8c005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:43:58.792 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.799+0000 7fa8ca7fc700 1 -- 172.21.15.23:0/1019634100 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fa8c4077fc0 con 0x7fa8d40f6770 2021-11-19T08:43:58.793 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.799+0000 7fa8d9a60700 1 --2- 172.21.15.23:0/1019634100 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa8c005c300 0x7fa8c005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:43:58.793 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.799+0000 7fa8d9a60700 1 --2- 172.21.15.23:0/1019634100 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa8c005c300 0x7fa8c005e7c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fa8d40f65d0 tx=0x7fa8d0009c00).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:43:58.793 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:58.803+0000 7fa8ca7fc700 1 -- 172.21.15.23:0/1019634100 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8c404a880 con 0x7fa8d40f6770 2021-11-19T08:43:58.986 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:59.002+0000 7fa8daa62700 1 -- 172.21.15.23:0/1019634100 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8d4045ad0 con 0x7fa8d40f6770 2021-11-19T08:43:58.987 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:59.002+0000 7fa8ca7fc700 1 -- 172.21.15.23:0/1019634100 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+2957 (secure 0 0 0) 0x7fa8d4045ad0 con 0x7fa8d40f6770 2021-11-19T08:43:58.988 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:43:58.988 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":74},{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":73,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:52.800166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:43:58.989 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:59.004+0000 7fa8daa62700 1 -- 172.21.15.23:0/1019634100 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa8c005c300 msgr2=0x7fa8c005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:58.989 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:59.004+0000 7fa8daa62700 1 --2- 172.21.15.23:0/1019634100 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa8c005c300 0x7fa8c005e7c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fa8d40f65d0 tx=0x7fa8d0009c00).stop 2021-11-19T08:43:58.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:59.004+0000 7fa8daa62700 1 -- 172.21.15.23:0/1019634100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8d40f6770 msgr2=0x7fa8d41061b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:43:58.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:59.004+0000 7fa8daa62700 1 --2- 172.21.15.23:0/1019634100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8d40f6770 0x7fa8d41061b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fa8c4009f50 tx=0x7fa8c4009b30).stop 2021-11-19T08:43:58.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:59.004+0000 7fa8daa62700 1 -- 172.21.15.23:0/1019634100 shutdown_connections 2021-11-19T08:43:58.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:59.004+0000 7fa8daa62700 1 --2- 172.21.15.23:0/1019634100 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8d40f54d0 0x7fa8d4105cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:58.991 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:59.004+0000 7fa8daa62700 1 --2- 172.21.15.23:0/1019634100 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa8c005c300 0x7fa8c005e7c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:58.991 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:59.004+0000 7fa8daa62700 1 --2- 172.21.15.23:0/1019634100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8d40f6770 0x7fa8d41061b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:43:58.991 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:59.004+0000 7fa8daa62700 1 -- 172.21.15.23:0/1019634100 >> 172.21.15.23:0/1019634100 conn(0x7fa8d40f0c30 msgr2=0x7fa8d40fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:43:58.991 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:59.004+0000 7fa8daa62700 1 -- 172.21.15.23:0/1019634100 shutdown_connections 2021-11-19T08:43:58.992 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:43:59.004+0000 7fa8daa62700 1 -- 172.21.15.23:0/1019634100 wait complete. 2021-11-19T08:43:58.992 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 74 2021-11-19T08:43:59.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:52.800166+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:clientreplay', 'state_seq': 17, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:43:59.001 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:clientreplay 2021-11-19T08:44:00.002 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:00.136 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.149+0000 7f8b70428700 1 -- 172.21.15.23:0/1616839087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b680f7d90 msgr2=0x7f8b680f81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:00.136 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.149+0000 7f8b70428700 1 --2- 172.21.15.23:0/1616839087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b680f7d90 0x7f8b680f81b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f8b600099e0 tx=0x7f8b60004d10).stop 2021-11-19T08:44:00.136 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.149+0000 7f8b70428700 1 -- 172.21.15.23:0/1616839087 shutdown_connections 2021-11-19T08:44:00.137 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.149+0000 7f8b70428700 1 --2- 172.21.15.23:0/1616839087 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b680f4780 0x7f8b680f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:00.137 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.149+0000 7f8b70428700 1 --2- 172.21.15.23:0/1616839087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b680f7d90 0x7f8b680f81b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:00.137 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.149+0000 7f8b70428700 1 -- 172.21.15.23:0/1616839087 >> 172.21.15.23:0/1616839087 conn(0x7f8b680f04a0 msgr2=0x7f8b680f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:00.137 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.149+0000 7f8b70428700 1 -- 172.21.15.23:0/1616839087 shutdown_connections 2021-11-19T08:44:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.149+0000 7f8b70428700 1 -- 172.21.15.23:0/1616839087 wait complete. 2021-11-19T08:44:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.150+0000 7f8b70428700 1 Processor -- start 2021-11-19T08:44:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.150+0000 7f8b70428700 1 -- start start 2021-11-19T08:44:00.138 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.150+0000 7f8b70428700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b680f4780 0x7f8b681070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.150+0000 7f8b70428700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b680f7d90 0x7f8b681075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.150+0000 7f8b70428700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b68107b50 con 0x7f8b680f7d90 2021-11-19T08:44:00.139 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.150+0000 7f8b70428700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b68107c90 con 0x7f8b680f4780 2021-11-19T08:44:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.150+0000 7f8b6d9c3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b680f7d90 0x7f8b681075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.150+0000 7f8b6d9c3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b680f7d90 0x7f8b681075a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48190/0 (socket says 172.21.15.23:48190) 2021-11-19T08:44:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.150+0000 7f8b6d9c3700 1 -- 172.21.15.23:0/3729162118 learned_addr learned my addr 172.21.15.23:0/3729162118 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:00.140 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.150+0000 7f8b6e1c4700 1 --2- 172.21.15.23:0/3729162118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b680f4780 0x7f8b681070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.151+0000 7f8b6d9c3700 1 -- 172.21.15.23:0/3729162118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b680f4780 msgr2=0x7f8b681070b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.151+0000 7f8b6d9c3700 1 --2- 172.21.15.23:0/3729162118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b680f4780 0x7f8b681070b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.151+0000 7f8b6d9c3700 1 -- 172.21.15.23:0/3729162118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b60009640 con 0x7f8b680f7d90 2021-11-19T08:44:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.151+0000 7f8b6d9c3700 1 --2- 172.21.15.23:0/3729162118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b680f7d90 0x7f8b681075a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f8b5c004f00 tx=0x7f8b5c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.151+0000 7f8b66ffd700 1 -- 172.21.15.23:0/3729162118 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b5c009500 con 0x7f8b680f7d90 2021-11-19T08:44:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.151+0000 7f8b70428700 1 -- 172.21.15.23:0/3729162118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b680f58f0 con 0x7f8b680f7d90 2021-11-19T08:44:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.151+0000 7f8b66ffd700 1 -- 172.21.15.23:0/3729162118 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b5c016070 con 0x7f8b680f7d90 2021-11-19T08:44:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.151+0000 7f8b66ffd700 1 -- 172.21.15.23:0/3729162118 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b5c012cb0 con 0x7f8b680f7d90 2021-11-19T08:44:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.151+0000 7f8b70428700 1 -- 172.21.15.23:0/3729162118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b680f5e30 con 0x7f8b680f7d90 2021-11-19T08:44:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.152+0000 7f8b70428700 1 -- 172.21.15.23:0/3729162118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b68027a00 con 0x7f8b680f7d90 2021-11-19T08:44:00.144 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.152+0000 7f8b66ffd700 1 -- 172.21.15.23:0/3729162118 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8b5c0077b0 con 0x7f8b680f7d90 2021-11-19T08:44:00.144 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.152+0000 7f8b66ffd700 1 --2- 172.21.15.23:0/3729162118 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8b5005c2a0 0x7f8b5005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:00.144 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.153+0000 7f8b66ffd700 1 -- 172.21.15.23:0/3729162118 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f8b5c07a300 con 0x7f8b680f7d90 2021-11-19T08:44:00.144 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.153+0000 7f8b6e1c4700 1 --2- 172.21.15.23:0/3729162118 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8b5005c2a0 0x7f8b5005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:00.145 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.153+0000 7f8b6e1c4700 1 --2- 172.21.15.23:0/3729162118 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8b5005c2a0 0x7f8b5005e760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8b6000b3b0 tx=0x7f8b60004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:00.145 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.156+0000 7f8b66ffd700 1 -- 172.21.15.23:0/3729162118 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b5c04cbc0 con 0x7f8b680f7d90 2021-11-19T08:44:00.289 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:59 smithi134 conmon[36363]: cluster 2021-11-19T08:43:58.882469+0000 mgr.smithi023.ehlpcl (mgr.14164) 299 : cluster 2021-11-19T08:44:00.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:59 smithi134 conmon[36363]: [DBG] pgmap v272: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 387 B/s wr, 1 op/s 2021-11-19T08:44:00.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:43:59 smithi134 conmon[36363]: audit 2021-11-19T08:43:59.003521+0000 mon.smithi023 (mon.0) 794 : audit [DBG] from='client.? 172.21.15.23:0/1019634100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:00.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.352+0000 7f8b70428700 1 -- 172.21.15.23:0/3729162118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b680fef40 con 0x7f8b680f7d90 2021-11-19T08:44:00.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.353+0000 7f8b66ffd700 1 -- 172.21.15.23:0/3729162118 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+2957 (secure 0 0 0) 0x7f8b5c020090 con 0x7f8b680f7d90 2021-11-19T08:44:00.338 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:00.338 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":74},{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":73,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:52.800166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:00.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.354+0000 7f8b70428700 1 -- 172.21.15.23:0/3729162118 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8b5005c2a0 msgr2=0x7f8b5005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:00.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.354+0000 7f8b70428700 1 --2- 172.21.15.23:0/3729162118 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8b5005c2a0 0x7f8b5005e760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8b6000b3b0 tx=0x7f8b60004d10).stop 2021-11-19T08:44:00.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.354+0000 7f8b70428700 1 -- 172.21.15.23:0/3729162118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b680f7d90 msgr2=0x7f8b681075a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:00.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.354+0000 7f8b70428700 1 --2- 172.21.15.23:0/3729162118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b680f7d90 0x7f8b681075a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f8b5c004f00 tx=0x7f8b5c009d80).stop 2021-11-19T08:44:00.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.354+0000 7f8b70428700 1 -- 172.21.15.23:0/3729162118 shutdown_connections 2021-11-19T08:44:00.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.354+0000 7f8b70428700 1 --2- 172.21.15.23:0/3729162118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b680f4780 0x7f8b681070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:00.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.354+0000 7f8b70428700 1 --2- 172.21.15.23:0/3729162118 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8b5005c2a0 0x7f8b5005e760 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:00.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.354+0000 7f8b70428700 1 --2- 172.21.15.23:0/3729162118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b680f7d90 0x7f8b681075a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:00.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.354+0000 7f8b70428700 1 -- 172.21.15.23:0/3729162118 >> 172.21.15.23:0/3729162118 conn(0x7f8b680f04a0 msgr2=0x7f8b680fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:00.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.354+0000 7f8b70428700 1 -- 172.21.15.23:0/3729162118 shutdown_connections 2021-11-19T08:44:00.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:00.354+0000 7f8b70428700 1 -- 172.21.15.23:0/3729162118 wait complete. 2021-11-19T08:44:00.342 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 74 2021-11-19T08:44:00.349 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:59 smithi023 conmon[33490]: cluster 2021-11-19T08:43:58.882469+0000 mgr.smithi023.ehlpcl (mgr.14164) 299 : cluster 2021-11-19T08:44:00.349 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:59 smithi023 conmon[33490]: [DBG] pgmap v272: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 387 B/s wr, 1 op/s 2021-11-19T08:44:00.350 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:43:59 smithi023 conmon[33490]: audit 2021-11-19T08:43:59.003521+0000 mon.smithi023 (mon.0) 794 : audit [DBG] from='client.? 172.21.15.23:0/1019634100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:00.351 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:52.800166+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:clientreplay', 'state_seq': 17, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:00.351 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:clientreplay 2021-11-19T08:44:01.249 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:00 smithi023 conmon[33490]: audit 2021-11-19T08:44:00.353701+0000 mon.smithi023 (mon.0) 795 : audit [DBG] from='client.? 172.21.15.23:0/3729162118' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:01.289 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:00 smithi134 conmon[36363]: audit 2021-11-19T08:44:00.353701+0000 mon.smithi023 (mon.0) 795 : audit [DBG] from='client.? 172.21.15.23:0/3729162118' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:01.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:00 smithi134 conmon[36363]: 2021-11-19T08:44:01.353 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.501+0000 7f37ec760700 1 -- 172.21.15.23:0/114844729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37e40f55a0 msgr2=0x7f37e40f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.501+0000 7f37ec760700 1 --2- 172.21.15.23:0/114844729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37e40f55a0 0x7f37e40f59c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f37e0004660 tx=0x7f37e0009b30).stop 2021-11-19T08:44:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.501+0000 7f37ec760700 1 -- 172.21.15.23:0/114844729 shutdown_connections 2021-11-19T08:44:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.501+0000 7f37ec760700 1 --2- 172.21.15.23:0/114844729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37e40f6840 0x7f37e40f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.501+0000 7f37ec760700 1 --2- 172.21.15.23:0/114844729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37e40f55a0 0x7f37e40f59c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.501+0000 7f37ec760700 1 -- 172.21.15.23:0/114844729 >> 172.21.15.23:0/114844729 conn(0x7f37e40f0ce0 msgr2=0x7f37e40f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.501+0000 7f37ec760700 1 -- 172.21.15.23:0/114844729 shutdown_connections 2021-11-19T08:44:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.501+0000 7f37ec760700 1 -- 172.21.15.23:0/114844729 wait complete. 2021-11-19T08:44:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37ec760700 1 Processor -- start 2021-11-19T08:44:01.492 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37ec760700 1 -- start start 2021-11-19T08:44:01.492 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37ec760700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37e40f55a0 0x7f37e4105dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:01.492 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37ec760700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37e40f6840 0x7f37e41062c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:01.492 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37ec760700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e4106870 con 0x7f37e40f6840 2021-11-19T08:44:01.493 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37ec760700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e41069b0 con 0x7f37e40f55a0 2021-11-19T08:44:01.493 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37e9cfb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37e40f6840 0x7f37e41062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:01.493 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37e9cfb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37e40f6840 0x7f37e41062c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48202/0 (socket says 172.21.15.23:48202) 2021-11-19T08:44:01.493 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37e9cfb700 1 -- 172.21.15.23:0/3560524411 learned_addr learned my addr 172.21.15.23:0/3560524411 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:01.494 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37ea4fc700 1 --2- 172.21.15.23:0/3560524411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37e40f55a0 0x7f37e4105dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:01.494 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37e9cfb700 1 -- 172.21.15.23:0/3560524411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37e40f55a0 msgr2=0x7f37e4105dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:01.494 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37e9cfb700 1 --2- 172.21.15.23:0/3560524411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37e40f55a0 0x7f37e4105dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:01.494 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37e9cfb700 1 -- 172.21.15.23:0/3560524411 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e0005040 con 0x7f37e40f6840 2021-11-19T08:44:01.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37e9cfb700 1 --2- 172.21.15.23:0/3560524411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37e40f6840 0x7f37e41062c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f37d400c530 tx=0x7f37d4007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:01.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.502+0000 7f37daffd700 1 -- 172.21.15.23:0/3560524411 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37d400f650 con 0x7f37e40f6840 2021-11-19T08:44:01.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.503+0000 7f37daffd700 1 -- 172.21.15.23:0/3560524411 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37d400fc60 con 0x7f37e40f6840 2021-11-19T08:44:01.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.503+0000 7f37daffd700 1 -- 172.21.15.23:0/3560524411 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37d4012c30 con 0x7f37e40f6840 2021-11-19T08:44:01.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.503+0000 7f37ec760700 1 -- 172.21.15.23:0/3560524411 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e410b450 con 0x7f37e40f6840 2021-11-19T08:44:01.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.503+0000 7f37ec760700 1 -- 172.21.15.23:0/3560524411 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e410b9c0 con 0x7f37e40f6840 2021-11-19T08:44:01.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.503+0000 7f37ec760700 1 -- 172.21.15.23:0/3560524411 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e4027a00 con 0x7f37e40f6840 2021-11-19T08:44:01.496 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.504+0000 7f37daffd700 1 -- 172.21.15.23:0/3560524411 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f37d4010070 con 0x7f37e40f6840 2021-11-19T08:44:01.497 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.504+0000 7f37daffd700 1 --2- 172.21.15.23:0/3560524411 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f37d005c2a0 0x7f37d005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:01.497 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.504+0000 7f37daffd700 1 -- 172.21.15.23:0/3560524411 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f37d4075f50 con 0x7f37e40f6840 2021-11-19T08:44:01.497 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.505+0000 7f37ea4fc700 1 --2- 172.21.15.23:0/3560524411 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f37d005c2a0 0x7f37d005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:01.497 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.505+0000 7f37ea4fc700 1 --2- 172.21.15.23:0/3560524411 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f37d005c2a0 0x7f37d005e760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f37e0004630 tx=0x7f37e0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:01.498 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.508+0000 7f37daffd700 1 -- 172.21.15.23:0/3560524411 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37d40452b0 con 0x7f37e40f6840 2021-11-19T08:44:01.691 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.706+0000 7f37ec760700 1 -- 172.21.15.23:0/3560524411 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37e40fdd30 con 0x7f37e40f6840 2021-11-19T08:44:01.691 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.706+0000 7f37daffd700 1 -- 172.21.15.23:0/3560524411 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+2957 (secure 0 0 0) 0x7f37d404c240 con 0x7f37e40f6840 2021-11-19T08:44:01.692 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:01.692 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":74},{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":73,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:52.800166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:01.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.708+0000 7f37ec760700 1 -- 172.21.15.23:0/3560524411 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f37d005c2a0 msgr2=0x7f37d005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:01.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.708+0000 7f37ec760700 1 --2- 172.21.15.23:0/3560524411 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f37d005c2a0 0x7f37d005e760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f37e0004630 tx=0x7f37e0009b30).stop 2021-11-19T08:44:01.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.708+0000 7f37ec760700 1 -- 172.21.15.23:0/3560524411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37e40f6840 msgr2=0x7f37e41062c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:01.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.708+0000 7f37ec760700 1 --2- 172.21.15.23:0/3560524411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37e40f6840 0x7f37e41062c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f37d400c530 tx=0x7f37d4007e20).stop 2021-11-19T08:44:01.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.708+0000 7f37ec760700 1 -- 172.21.15.23:0/3560524411 shutdown_connections 2021-11-19T08:44:01.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.708+0000 7f37ec760700 1 --2- 172.21.15.23:0/3560524411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37e40f55a0 0x7f37e4105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:01.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.708+0000 7f37ec760700 1 --2- 172.21.15.23:0/3560524411 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f37d005c2a0 0x7f37d005e760 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:01.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.708+0000 7f37ec760700 1 --2- 172.21.15.23:0/3560524411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37e40f6840 0x7f37e41062c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:01.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.708+0000 7f37ec760700 1 -- 172.21.15.23:0/3560524411 >> 172.21.15.23:0/3560524411 conn(0x7f37e40f0ce0 msgr2=0x7f37e40fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:01.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.708+0000 7f37ec760700 1 -- 172.21.15.23:0/3560524411 shutdown_connections 2021-11-19T08:44:01.699 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:01.708+0000 7f37ec760700 1 -- 172.21.15.23:0/3560524411 wait complete. 2021-11-19T08:44:01.699 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 74 2021-11-19T08:44:01.704 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:52.800166+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:clientreplay', 'state_seq': 17, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:01.705 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:clientreplay 2021-11-19T08:44:02.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:01 smithi023 conmon[33490]: cluster 2021-11-19T08:44:00.883006+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:44:02.141 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:01 smithi023 conmon[33490]: mgr.14164) 300 : cluster [DBG] pgmap v273: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-19T08:44:02.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:01 smithi023 conmon[33490]: audit 2021-11-19T08:44:01.707132+0000 mon.smithi023 ( 2021-11-19T08:44:02.142 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:01 smithi023 conmon[33490]: mon.0) 796 : audit [DBG] from='client.? 172.21.15.23:0/3560524411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:02.289 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:01 smithi134 conmon[36363]: cluster 2021-11-19T08:44:00.883006+0000 mgr.smithi023.ehlpcl (mgr.14164) 300 : cluster 2021-11-19T08:44:02.289 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:01 smithi134 conmon[36363]: [DBG] pgmap v273: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 341 B/s wr, 1 op/s 2021-11-19T08:44:02.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:01 smithi134 conmon[36363]: audit 2021-11-19T08:44:01.707132+0000 mon.smithi023 (mon.0) 796 : audit [DBG] from='client.? 172.21.15.23:0/3560524411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:02.706 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.854+0000 7f17b988f700 1 -- 172.21.15.23:0/1335085627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17b40f7d90 msgr2=0x7f17b40f81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.854+0000 7f17b988f700 1 --2- 172.21.15.23:0/1335085627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17b40f7d90 0x7f17b40f81b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f17a4004660 tx=0x7f17a4009b30).stop 2021-11-19T08:44:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.854+0000 7f17b988f700 1 -- 172.21.15.23:0/1335085627 shutdown_connections 2021-11-19T08:44:02.844 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.854+0000 7f17b988f700 1 --2- 172.21.15.23:0/1335085627 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17b40f4780 0x7f17b40f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:02.844 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.854+0000 7f17b988f700 1 --2- 172.21.15.23:0/1335085627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17b40f7d90 0x7f17b40f81b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:02.844 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.854+0000 7f17b988f700 1 -- 172.21.15.23:0/1335085627 >> 172.21.15.23:0/1335085627 conn(0x7f17b40f04a0 msgr2=0x7f17b40f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:02.844 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.854+0000 7f17b988f700 1 -- 172.21.15.23:0/1335085627 shutdown_connections 2021-11-19T08:44:02.844 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.854+0000 7f17b988f700 1 -- 172.21.15.23:0/1335085627 wait complete. 2021-11-19T08:44:02.845 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.855+0000 7f17b988f700 1 Processor -- start 2021-11-19T08:44:02.845 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.855+0000 7f17b988f700 1 -- start start 2021-11-19T08:44:02.845 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.855+0000 7f17b988f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17b40f4780 0x7f17b40f7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:02.845 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.855+0000 7f17b988f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17b40f7d90 0x7f17b40f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:02.846 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.855+0000 7f17b988f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b40f7900 con 0x7f17b40f7d90 2021-11-19T08:44:02.846 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.855+0000 7f17b988f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b40f5ea0 con 0x7f17b40f4780 2021-11-19T08:44:02.846 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.855+0000 7f17b27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17b40f7d90 0x7f17b40f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:02.846 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.855+0000 7f17b27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17b40f7d90 0x7f17b40f59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48210/0 (socket says 172.21.15.23:48210) 2021-11-19T08:44:02.847 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.855+0000 7f17b27fc700 1 -- 172.21.15.23:0/1885085700 learned_addr learned my addr 172.21.15.23:0/1885085700 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:02.847 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.855+0000 7f17b2ffd700 1 --2- 172.21.15.23:0/1885085700 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17b40f4780 0x7f17b40f7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:02.847 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.856+0000 7f17b27fc700 1 -- 172.21.15.23:0/1885085700 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17b40f4780 msgr2=0x7f17b40f7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:02.847 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.856+0000 7f17b27fc700 1 --2- 172.21.15.23:0/1885085700 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17b40f4780 0x7f17b40f7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:02.847 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.856+0000 7f17b27fc700 1 -- 172.21.15.23:0/1885085700 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a4005040 con 0x7f17b40f7d90 2021-11-19T08:44:02.848 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.856+0000 7f17b27fc700 1 --2- 172.21.15.23:0/1885085700 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17b40f7d90 0x7f17b40f59b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f179c002700 tx=0x7f179c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:02.848 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.856+0000 7f17ab7fe700 1 -- 172.21.15.23:0/1885085700 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f179c0079c0 con 0x7f17b40f7d90 2021-11-19T08:44:02.848 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.856+0000 7f17ab7fe700 1 -- 172.21.15.23:0/1885085700 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f179c00f040 con 0x7f17b40f7d90 2021-11-19T08:44:02.849 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.856+0000 7f17b988f700 1 -- 172.21.15.23:0/1885085700 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b40f6120 con 0x7f17b40f7d90 2021-11-19T08:44:02.849 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.856+0000 7f17ab7fe700 1 -- 172.21.15.23:0/1885085700 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f179c014860 con 0x7f17b40f7d90 2021-11-19T08:44:02.850 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.856+0000 7f17b988f700 1 -- 172.21.15.23:0/1885085700 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b40f66e0 con 0x7f17b40f7d90 2021-11-19T08:44:02.850 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.857+0000 7f17b988f700 1 -- 172.21.15.23:0/1885085700 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b4027a00 con 0x7f17b40f7d90 2021-11-19T08:44:02.850 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.857+0000 7f17ab7fe700 1 -- 172.21.15.23:0/1885085700 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f179c01b060 con 0x7f17b40f7d90 2021-11-19T08:44:02.850 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.858+0000 7f17ab7fe700 1 --2- 172.21.15.23:0/1885085700 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f17a005c2a0 0x7f17a005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:02.851 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.858+0000 7f17ab7fe700 1 -- 172.21.15.23:0/1885085700 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f179c0760e0 con 0x7f17b40f7d90 2021-11-19T08:44:02.851 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.858+0000 7f17b2ffd700 1 --2- 172.21.15.23:0/1885085700 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f17a005c2a0 0x7f17a005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:02.851 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.858+0000 7f17b2ffd700 1 --2- 172.21.15.23:0/1885085700 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f17a005c2a0 0x7f17a005e760 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f17a4009f50 tx=0x7f17a4009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:02.852 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:02.862+0000 7f17ab7fe700 1 -- 172.21.15.23:0/1885085700 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f179c048a20 con 0x7f17b40f7d90 2021-11-19T08:44:03.045 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:03.060+0000 7f17b988f700 1 -- 172.21.15.23:0/1885085700 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17b40fef40 con 0x7f17b40f7d90 2021-11-19T08:44:03.045 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:03.061+0000 7f17ab7fe700 1 -- 172.21.15.23:0/1885085700 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+2957 (secure 0 0 0) 0x7f179c04c040 con 0x7f17b40f7d90 2021-11-19T08:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:03.046 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":74},{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":73,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:52.800166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:03.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:03.062+0000 7f17b988f700 1 -- 172.21.15.23:0/1885085700 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f17a005c2a0 msgr2=0x7f17a005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:03.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:03.062+0000 7f17b988f700 1 --2- 172.21.15.23:0/1885085700 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f17a005c2a0 0x7f17a005e760 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f17a4009f50 tx=0x7f17a4009b30).stop 2021-11-19T08:44:03.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:03.062+0000 7f17b988f700 1 -- 172.21.15.23:0/1885085700 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17b40f7d90 msgr2=0x7f17b40f59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:03.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:03.062+0000 7f17b988f700 1 --2- 172.21.15.23:0/1885085700 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17b40f7d90 0x7f17b40f59b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f179c002700 tx=0x7f179c009d80).stop 2021-11-19T08:44:03.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:03.062+0000 7f17b988f700 1 -- 172.21.15.23:0/1885085700 shutdown_connections 2021-11-19T08:44:03.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:03.062+0000 7f17b988f700 1 --2- 172.21.15.23:0/1885085700 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17b40f4780 0x7f17b40f7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:03.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:03.062+0000 7f17b988f700 1 --2- 172.21.15.23:0/1885085700 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f17a005c2a0 0x7f17a005e760 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:03.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:03.062+0000 7f17b988f700 1 --2- 172.21.15.23:0/1885085700 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17b40f7d90 0x7f17b40f59b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:03.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:03.062+0000 7f17b988f700 1 -- 172.21.15.23:0/1885085700 >> 172.21.15.23:0/1885085700 conn(0x7f17b40f04a0 msgr2=0x7f17b40fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:03.050 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:03.062+0000 7f17b988f700 1 -- 172.21.15.23:0/1885085700 shutdown_connections 2021-11-19T08:44:03.050 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:03.062+0000 7f17b988f700 1 -- 172.21.15.23:0/1885085700 wait complete. 2021-11-19T08:44:03.050 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 74 2021-11-19T08:44:03.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:52.800166+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:clientreplay', 'state_seq': 17, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:03.059 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:clientreplay 2021-11-19T08:44:04.061 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:04.197 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.209+0000 7f4017fc1700 1 -- 172.21.15.23:0/4060609216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40100f6680 msgr2=0x7f40100f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:04.198 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.209+0000 7f4017fc1700 1 --2- 172.21.15.23:0/4060609216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40100f6680 0x7f40100f6aa0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f40080099e0 tx=0x7f4008004d10).stop 2021-11-19T08:44:04.198 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.209+0000 7f4017fc1700 1 -- 172.21.15.23:0/4060609216 shutdown_connections 2021-11-19T08:44:04.198 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.209+0000 7f4017fc1700 1 --2- 172.21.15.23:0/4060609216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40100f7920 0x7f40100f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:04.199 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.209+0000 7f4017fc1700 1 --2- 172.21.15.23:0/4060609216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40100f6680 0x7f40100f6aa0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:04.199 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.209+0000 7f4017fc1700 1 -- 172.21.15.23:0/4060609216 >> 172.21.15.23:0/4060609216 conn(0x7f40100f1de0 msgr2=0x7f40100f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:04.199 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.209+0000 7f4017fc1700 1 -- 172.21.15.23:0/4060609216 shutdown_connections 2021-11-19T08:44:04.200 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.209+0000 7f4017fc1700 1 -- 172.21.15.23:0/4060609216 wait complete. 2021-11-19T08:44:04.200 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.210+0000 7f4017fc1700 1 Processor -- start 2021-11-19T08:44:04.200 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.210+0000 7f4017fc1700 1 -- start start 2021-11-19T08:44:04.200 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.210+0000 7f4017fc1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40100f6680 0x7f4010106f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:04.200 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.210+0000 7f4017fc1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40100f7920 0x7f4010107480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:04.201 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.210+0000 7f4017fc1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4010107a30 con 0x7f40100f7920 2021-11-19T08:44:04.201 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.210+0000 7f4017fc1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4010107b70 con 0x7f40100f6680 2021-11-19T08:44:04.201 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.210+0000 7f401555c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40100f7920 0x7f4010107480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:04.201 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.210+0000 7f401555c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40100f7920 0x7f4010107480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48220/0 (socket says 172.21.15.23:48220) 2021-11-19T08:44:04.202 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.210+0000 7f401555c700 1 -- 172.21.15.23:0/1925299424 learned_addr learned my addr 172.21.15.23:0/1925299424 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:04.202 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.210+0000 7f4015d5d700 1 --2- 172.21.15.23:0/1925299424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40100f6680 0x7f4010106f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:04.202 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.210+0000 7f401555c700 1 -- 172.21.15.23:0/1925299424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40100f6680 msgr2=0x7f4010106f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:04.202 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.210+0000 7f401555c700 1 --2- 172.21.15.23:0/1925299424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40100f6680 0x7f4010106f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:04.202 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.210+0000 7f401555c700 1 -- 172.21.15.23:0/1925299424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4008009640 con 0x7f40100f7920 2021-11-19T08:44:04.203 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.210+0000 7f401555c700 1 --2- 172.21.15.23:0/1925299424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40100f7920 0x7f4010107480 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f400400c790 tx=0x7f4004007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:04.203 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.211+0000 7f400e7fc700 1 -- 172.21.15.23:0/1925299424 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f400400e680 con 0x7f40100f7920 2021-11-19T08:44:04.203 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.211+0000 7f4017fc1700 1 -- 172.21.15.23:0/1925299424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401010c610 con 0x7f40100f7920 2021-11-19T08:44:04.203 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.211+0000 7f400e7fc700 1 -- 172.21.15.23:0/1925299424 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f400400ec90 con 0x7f40100f7920 2021-11-19T08:44:04.204 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.211+0000 7f400e7fc700 1 -- 172.21.15.23:0/1925299424 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4004013d60 con 0x7f40100f7920 2021-11-19T08:44:04.204 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.211+0000 7f4017fc1700 1 -- 172.21.15.23:0/1925299424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401010cb50 con 0x7f40100f7920 2021-11-19T08:44:04.204 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.212+0000 7f4017fc1700 1 -- 172.21.15.23:0/1925299424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4010027a00 con 0x7f40100f7920 2021-11-19T08:44:04.204 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.212+0000 7f400e7fc700 1 -- 172.21.15.23:0/1925299424 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f400401a060 con 0x7f40100f7920 2021-11-19T08:44:04.204 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.212+0000 7f400e7fc700 1 --2- 172.21.15.23:0/1925299424 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3ff805c2a0 0x7f3ff805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:04.205 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.212+0000 7f400e7fc700 1 -- 172.21.15.23:0/1925299424 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f4004076520 con 0x7f40100f7920 2021-11-19T08:44:04.205 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.212+0000 7f4015d5d700 1 --2- 172.21.15.23:0/1925299424 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3ff805c2a0 0x7f3ff805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:04.205 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.213+0000 7f4015d5d700 1 --2- 172.21.15.23:0/1925299424 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3ff805c2a0 0x7f3ff805e760 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f4008000f80 tx=0x7f4008004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:04.206 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.216+0000 7f400e7fc700 1 -- 172.21.15.23:0/1925299424 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4004048de0 con 0x7f40100f7920 2021-11-19T08:44:04.289 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:03 smithi134 conmon[36363]: cluster 2021-11-19T08:44:02.883357+0000 mgr.smithi023.ehlpcl (mgr.14164) 301 : cluster [DBG] pgmap v274: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 341 B/s wr, 0 op/s 2021-11-19T08:44:04.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:03 smithi134 conmon[36363]: audit 2021-11-19T08 2021-11-19T08:44:04.290 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:03 smithi134 conmon[36363]: :44:03.061654+0000 mon.smithi023 (mon.0) 797 : audit [DBG] from='client.? 172.21.15.23:0/1885085700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:04.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:03 smithi023 conmon[33490]: cluster 2021-11-19T08:44:02.883357+0000 mgr.smithi023.ehlpcl 2021-11-19T08:44:04.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:03 smithi023 conmon[33490]: (mgr.14164) 301 : cluster [DBG] pgmap v274: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 341 B/s wr, 0 op/s 2021-11-19T08:44:04.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:03 smithi023 conmon[33490]: audit 2021-11-19T08:44: 2021-11-19T08:44:04.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:03 smithi023 conmon[33490]: 03.061654+0000 mon.smithi023 (mon.0) 797 : audit [DBG] from='client.? 172.21.15.23:0/1885085700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:04.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:03 smithi023 conmon[33490]: 2021-11-19T08:44:04.397 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.413+0000 7f4017fc1700 1 -- 172.21.15.23:0/1925299424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40100fee10 con 0x7f40100f7920 2021-11-19T08:44:04.397 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.413+0000 7f400e7fc700 1 -- 172.21.15.23:0/1925299424 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+2957 (secure 0 0 0) 0x7f400404c400 con 0x7f40100f7920 2021-11-19T08:44:04.398 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:04.398 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":74},{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":73,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:52.800166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:04.400 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.415+0000 7f4017fc1700 1 -- 172.21.15.23:0/1925299424 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3ff805c2a0 msgr2=0x7f3ff805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:04.400 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.415+0000 7f4017fc1700 1 --2- 172.21.15.23:0/1925299424 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3ff805c2a0 0x7f3ff805e760 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f4008000f80 tx=0x7f4008004d10).stop 2021-11-19T08:44:04.400 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.415+0000 7f4017fc1700 1 -- 172.21.15.23:0/1925299424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40100f7920 msgr2=0x7f4010107480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:04.401 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.415+0000 7f4017fc1700 1 --2- 172.21.15.23:0/1925299424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40100f7920 0x7f4010107480 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f400400c790 tx=0x7f4004007d70).stop 2021-11-19T08:44:04.401 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.415+0000 7f4017fc1700 1 -- 172.21.15.23:0/1925299424 shutdown_connections 2021-11-19T08:44:04.401 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.415+0000 7f4017fc1700 1 --2- 172.21.15.23:0/1925299424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f40100f6680 0x7f4010106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:04.401 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.415+0000 7f4017fc1700 1 --2- 172.21.15.23:0/1925299424 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f3ff805c2a0 0x7f3ff805e760 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:04.401 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.415+0000 7f4017fc1700 1 --2- 172.21.15.23:0/1925299424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40100f7920 0x7f4010107480 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:04.402 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.415+0000 7f4017fc1700 1 -- 172.21.15.23:0/1925299424 >> 172.21.15.23:0/1925299424 conn(0x7f40100f1de0 msgr2=0x7f40100fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:04.402 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.415+0000 7f4017fc1700 1 -- 172.21.15.23:0/1925299424 shutdown_connections 2021-11-19T08:44:04.402 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:04.415+0000 7f4017fc1700 1 -- 172.21.15.23:0/1925299424 wait complete. 2021-11-19T08:44:04.402 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 74 2021-11-19T08:44:04.411 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:52.800166+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:clientreplay', 'state_seq': 17, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:04.412 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:clientreplay 2021-11-19T08:44:05.412 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:05.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:05 smithi134 conmon[36363]: audit 2021-11-19T08:44:04.414346+0000 mon.smithi023 (mon.0) 798 : audit 2021-11-19T08:44:05.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:05 smithi134 conmon[36363]: [DBG] from='client.? 172.21.15.23:0/1925299424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:05.553 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.563+0000 7ff6e17d8700 1 -- 172.21.15.23:0/1498004941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6dc0f65b0 msgr2=0x7ff6dc0f69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:05.553 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.563+0000 7ff6e17d8700 1 --2- 172.21.15.23:0/1498004941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6dc0f65b0 0x7ff6dc0f69d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7ff6d4004660 tx=0x7ff6d4009b30).stop 2021-11-19T08:44:05.554 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.563+0000 7ff6e17d8700 1 -- 172.21.15.23:0/1498004941 shutdown_connections 2021-11-19T08:44:05.554 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.563+0000 7ff6e17d8700 1 --2- 172.21.15.23:0/1498004941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6dc0f7850 0x7ff6dc0f7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:05.555 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.563+0000 7ff6e17d8700 1 --2- 172.21.15.23:0/1498004941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6dc0f65b0 0x7ff6dc0f69d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:05.555 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.563+0000 7ff6e17d8700 1 -- 172.21.15.23:0/1498004941 >> 172.21.15.23:0/1498004941 conn(0x7ff6dc0f1d10 msgr2=0x7ff6dc0f4130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:05.555 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.563+0000 7ff6e17d8700 1 -- 172.21.15.23:0/1498004941 shutdown_connections 2021-11-19T08:44:05.556 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.563+0000 7ff6e17d8700 1 -- 172.21.15.23:0/1498004941 wait complete. 2021-11-19T08:44:05.556 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.563+0000 7ff6e17d8700 1 Processor -- start 2021-11-19T08:44:05.557 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.563+0000 7ff6e17d8700 1 -- start start 2021-11-19T08:44:05.557 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6e17d8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6dc0f65b0 0x7ff6dc106e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:05.557 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6e17d8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6dc0f7850 0x7ff6dc107300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:05.558 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6e17d8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc107900 con 0x7ff6dc0f65b0 2021-11-19T08:44:05.558 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6e17d8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc108110 con 0x7ff6dc0f7850 2021-11-19T08:44:05.559 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6daffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6dc0f65b0 0x7ff6dc106e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:05.559 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6daffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6dc0f65b0 0x7ff6dc106e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48228/0 (socket says 172.21.15.23:48228) 2021-11-19T08:44:05.560 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6daffd700 1 -- 172.21.15.23:0/3350986769 learned_addr learned my addr 172.21.15.23:0/3350986769 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:05.560 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6da7fc700 1 --2- 172.21.15.23:0/3350986769 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6dc0f7850 0x7ff6dc107300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:05.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6daffd700 1 -- 172.21.15.23:0/3350986769 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6dc0f7850 msgr2=0x7ff6dc107300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:05.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6daffd700 1 --2- 172.21.15.23:0/3350986769 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6dc0f7850 0x7ff6dc107300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:05.562 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6daffd700 1 -- 172.21.15.23:0/3350986769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d4005040 con 0x7ff6dc0f65b0 2021-11-19T08:44:05.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6da7fc700 1 --2- 172.21.15.23:0/3350986769 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6dc0f7850 0x7ff6dc107300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:44:05.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6daffd700 1 --2- 172.21.15.23:0/3350986769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6dc0f65b0 0x7ff6dc106e10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7ff6d4009f50 tx=0x7ff6d4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:05.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6d37fe700 1 -- 172.21.15.23:0/3350986769 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6d4018070 con 0x7ff6dc0f65b0 2021-11-19T08:44:05.564 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6e17d8700 1 -- 172.21.15.23:0/3350986769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6dc108330 con 0x7ff6dc0f65b0 2021-11-19T08:44:05.564 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6d37fe700 1 -- 172.21.15.23:0/3350986769 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6d4013410 con 0x7ff6dc0f65b0 2021-11-19T08:44:05.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.564+0000 7ff6d37fe700 1 -- 172.21.15.23:0/3350986769 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6d401c620 con 0x7ff6dc0f65b0 2021-11-19T08:44:05.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.565+0000 7ff6e17d8700 1 -- 172.21.15.23:0/3350986769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6dc108890 con 0x7ff6dc0f65b0 2021-11-19T08:44:05.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.565+0000 7ff6e17d8700 1 -- 172.21.15.23:0/3350986769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6dc0f59a0 con 0x7ff6dc0f65b0 2021-11-19T08:44:05.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.566+0000 7ff6d37fe700 1 -- 172.21.15.23:0/3350986769 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff6d401d420 con 0x7ff6dc0f65b0 2021-11-19T08:44:05.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.566+0000 7ff6d37fe700 1 --2- 172.21.15.23:0/3350986769 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff6c805c300 0x7ff6c805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:05.567 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.566+0000 7ff6d37fe700 1 -- 172.21.15.23:0/3350986769 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7ff6d4078240 con 0x7ff6dc0f65b0 2021-11-19T08:44:05.567 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.566+0000 7ff6da7fc700 1 --2- 172.21.15.23:0/3350986769 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff6c805c300 0x7ff6c805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:05.568 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.567+0000 7ff6da7fc700 1 --2- 172.21.15.23:0/3350986769 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff6c805c300 0x7ff6c805e7c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7ff6c4004a90 tx=0x7ff6c4009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:05.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.570+0000 7ff6d37fe700 1 -- 172.21.15.23:0/3350986769 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6d404aa40 con 0x7ff6dc0f65b0 2021-11-19T08:44:05.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:05 smithi023 conmon[33490]: audit 2021-11-19T08:44:04.414346+0000 mon.smithi023 (mon. 2021-11-19T08:44:05.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:05 smithi023 conmon[33490]: 0) 798 : audit [DBG] from='client.? 172.21.15.23:0/1925299424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:05.749 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.765+0000 7ff6e17d8700 1 -- 172.21.15.23:0/3350986769 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6dc0fed40 con 0x7ff6dc0f65b0 2021-11-19T08:44:05.750 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.765+0000 7ff6d37fe700 1 -- 172.21.15.23:0/3350986769 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+2957 (secure 0 0 0) 0x7ff6d404e060 con 0x7ff6dc0f65b0 2021-11-19T08:44:05.751 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:05.751 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":74},{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":73,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:52.800166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.767+0000 7ff6e17d8700 1 -- 172.21.15.23:0/3350986769 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff6c805c300 msgr2=0x7ff6c805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.767+0000 7ff6e17d8700 1 --2- 172.21.15.23:0/3350986769 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff6c805c300 0x7ff6c805e7c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7ff6c4004a90 tx=0x7ff6c4009040).stop 2021-11-19T08:44:05.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.767+0000 7ff6e17d8700 1 -- 172.21.15.23:0/3350986769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6dc0f65b0 msgr2=0x7ff6dc106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.767+0000 7ff6e17d8700 1 --2- 172.21.15.23:0/3350986769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6dc0f65b0 0x7ff6dc106e10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7ff6d4009f50 tx=0x7ff6d4009b30).stop 2021-11-19T08:44:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.767+0000 7ff6e17d8700 1 -- 172.21.15.23:0/3350986769 shutdown_connections 2021-11-19T08:44:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.767+0000 7ff6e17d8700 1 --2- 172.21.15.23:0/3350986769 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff6c805c300 0x7ff6c805e7c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.767+0000 7ff6e17d8700 1 --2- 172.21.15.23:0/3350986769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6dc0f65b0 0x7ff6dc106e10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:05.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.767+0000 7ff6e17d8700 1 --2- 172.21.15.23:0/3350986769 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6dc0f7850 0x7ff6dc107300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:05.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.767+0000 7ff6e17d8700 1 -- 172.21.15.23:0/3350986769 >> 172.21.15.23:0/3350986769 conn(0x7ff6dc0f1d10 msgr2=0x7ff6dc0fb2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:05.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.767+0000 7ff6e17d8700 1 -- 172.21.15.23:0/3350986769 shutdown_connections 2021-11-19T08:44:05.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:05.767+0000 7ff6e17d8700 1 -- 172.21.15.23:0/3350986769 wait complete. 2021-11-19T08:44:05.755 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 74 2021-11-19T08:44:05.763 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:52.800166+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:clientreplay', 'state_seq': 17, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:05.764 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:clientreplay 2021-11-19T08:44:06.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:06 smithi023 conmon[33490]: cluster 2021-11-19T08:44:04.884094+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:44:06.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:06 smithi023 conmon[33490]: mgr.14164) 302 : cluster [DBG] pgmap v275: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 341 B/s wr, 0 op/s 2021-11-19T08:44:06.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:06 smithi023 conmon[33490]: audit 2021-11-19T08:44:05.766542+0000 mon.smithi023 (mon.0) 799 : audit [DBG] from='client.? 172.21.15.23:0/3350986769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:06.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:06 smithi134 conmon[36363]: cluster 2021-11-19T08:44:04.884094+0000 mgr.smithi023.ehlpcl (mgr.14164) 302 : cluster [DBG] pgmap v275: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 341 B/s wr, 0 op/s 2021-11-19T08:44:06.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:06 smithi134 conmon[36363]: audit 2021-11-19T08:44 2021-11-19T08:44:06.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:06 smithi134 conmon[36363]: :05.766542+0000 mon.smithi023 (mon.0) 799 : audit [DBG] from='client.? 172.21.15.23:0/3350986769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:06.765 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:06.898 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.912+0000 7ff7908b7700 1 -- 172.21.15.23:0/3218407929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f6680 msgr2=0x7ff78c0f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:06.899 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.912+0000 7ff7908b7700 1 --2- 172.21.15.23:0/3218407929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f6680 0x7ff78c0f6aa0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7ff77c004660 tx=0x7ff77c009b30).stop 2021-11-19T08:44:06.899 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.912+0000 7ff7908b7700 1 -- 172.21.15.23:0/3218407929 shutdown_connections 2021-11-19T08:44:06.899 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.912+0000 7ff7908b7700 1 --2- 172.21.15.23:0/3218407929 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f7920 0x7ff78c0f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:06.900 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.912+0000 7ff7908b7700 1 --2- 172.21.15.23:0/3218407929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f6680 0x7ff78c0f6aa0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:06.900 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.912+0000 7ff7908b7700 1 -- 172.21.15.23:0/3218407929 >> 172.21.15.23:0/3218407929 conn(0x7ff78c0f1de0 msgr2=0x7ff78c0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:06.900 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.912+0000 7ff7908b7700 1 -- 172.21.15.23:0/3218407929 shutdown_connections 2021-11-19T08:44:06.900 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.912+0000 7ff7908b7700 1 -- 172.21.15.23:0/3218407929 wait complete. 2021-11-19T08:44:06.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.913+0000 7ff7908b7700 1 Processor -- start 2021-11-19T08:44:06.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.913+0000 7ff7908b7700 1 -- start start 2021-11-19T08:44:06.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.913+0000 7ff7908b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f6680 0x7ff78c106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:06.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.913+0000 7ff7908b7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f7920 0x7ff78c1073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:06.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.913+0000 7ff7908b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c107950 con 0x7ff78c0f6680 2021-11-19T08:44:06.902 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.913+0000 7ff7908b7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c107a90 con 0x7ff78c0f7920 2021-11-19T08:44:06.902 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.913+0000 7ff78a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f6680 0x7ff78c106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:06.902 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.913+0000 7ff78a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f6680 0x7ff78c106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48236/0 (socket says 172.21.15.23:48236) 2021-11-19T08:44:06.902 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.913+0000 7ff78a59c700 1 -- 172.21.15.23:0/1842234674 learned_addr learned my addr 172.21.15.23:0/1842234674 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:06.902 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.913+0000 7ff789d9b700 1 --2- 172.21.15.23:0/1842234674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f7920 0x7ff78c1073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.913+0000 7ff78a59c700 1 -- 172.21.15.23:0/1842234674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f7920 msgr2=0x7ff78c1073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.913+0000 7ff78a59c700 1 --2- 172.21.15.23:0/1842234674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f7920 0x7ff78c1073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.913+0000 7ff78a59c700 1 -- 172.21.15.23:0/1842234674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c005040 con 0x7ff78c0f6680 2021-11-19T08:44:06.903 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.913+0000 7ff78a59c700 1 --2- 172.21.15.23:0/1842234674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f6680 0x7ff78c106eb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7ff77c004e30 tx=0x7ff77c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:06.904 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.914+0000 7ff782ffd700 1 -- 172.21.15.23:0/1842234674 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff77c018070 con 0x7ff78c0f6680 2021-11-19T08:44:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.914+0000 7ff782ffd700 1 -- 172.21.15.23:0/1842234674 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff77c013410 con 0x7ff78c0f6680 2021-11-19T08:44:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.914+0000 7ff782ffd700 1 -- 172.21.15.23:0/1842234674 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff77c01c3c0 con 0x7ff78c0f6680 2021-11-19T08:44:06.905 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.914+0000 7ff7908b7700 1 -- 172.21.15.23:0/1842234674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78c10c4d0 con 0x7ff78c0f6680 2021-11-19T08:44:06.906 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.914+0000 7ff7908b7700 1 -- 172.21.15.23:0/1842234674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c10ca90 con 0x7ff78c0f6680 2021-11-19T08:44:06.907 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.915+0000 7ff7908b7700 1 -- 172.21.15.23:0/1842234674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff78c027a00 con 0x7ff78c0f6680 2021-11-19T08:44:06.907 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.915+0000 7ff782ffd700 1 -- 172.21.15.23:0/1842234674 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff77c01d420 con 0x7ff78c0f6680 2021-11-19T08:44:06.907 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.915+0000 7ff782ffd700 1 --2- 172.21.15.23:0/1842234674 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff77805c2a0 0x7ff77805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:06.907 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.915+0000 7ff782ffd700 1 -- 172.21.15.23:0/1842234674 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6414+0+0 (secure 0 0 0) 0x7ff77c077f60 con 0x7ff78c0f6680 2021-11-19T08:44:06.908 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.915+0000 7ff789d9b700 1 --2- 172.21.15.23:0/1842234674 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff77805c2a0 0x7ff77805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:06.908 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.916+0000 7ff789d9b700 1 --2- 172.21.15.23:0/1842234674 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff77805c2a0 0x7ff77805e760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7ff7740096c0 tx=0x7ff774009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:06.908 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:06.919+0000 7ff782ffd700 1 -- 172.21.15.23:0/1842234674 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff77c0a42b0 con 0x7ff78c0f6680 2021-11-19T08:44:07.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:07.120+0000 7ff7908b7700 1 -- 172.21.15.23:0/1842234674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff78c0fee10 con 0x7ff78c0f6680 2021-11-19T08:44:07.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:07.120+0000 7ff782ffd700 1 -- 172.21.15.23:0/1842234674 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+2957 (secure 0 0 0) 0x7ff77c04a820 con 0x7ff78c0f6680 2021-11-19T08:44:07.105 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:07.106 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":74},{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":73,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:43:52.800166+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24411,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24411":{"gid":24411,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":65,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.134:6825/464801254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":464801254},{"type":"v1","addr":"172.21.15.134:6825","nonce":464801254}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:07.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:07.122+0000 7ff7908b7700 1 -- 172.21.15.23:0/1842234674 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff77805c2a0 msgr2=0x7ff77805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:07.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:07.122+0000 7ff7908b7700 1 --2- 172.21.15.23:0/1842234674 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff77805c2a0 0x7ff77805e760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7ff7740096c0 tx=0x7ff774009be0).stop 2021-11-19T08:44:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:07.122+0000 7ff7908b7700 1 -- 172.21.15.23:0/1842234674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f6680 msgr2=0x7ff78c106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:07.122+0000 7ff7908b7700 1 --2- 172.21.15.23:0/1842234674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f6680 0x7ff78c106eb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7ff77c004e30 tx=0x7ff77c009b30).stop 2021-11-19T08:44:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:07.122+0000 7ff7908b7700 1 -- 172.21.15.23:0/1842234674 shutdown_connections 2021-11-19T08:44:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:07.122+0000 7ff7908b7700 1 --2- 172.21.15.23:0/1842234674 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7ff77805c2a0 0x7ff77805e760 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:07.122+0000 7ff7908b7700 1 --2- 172.21.15.23:0/1842234674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c0f6680 0x7ff78c106eb0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:07.122+0000 7ff7908b7700 1 --2- 172.21.15.23:0/1842234674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0f7920 0x7ff78c1073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:07.122+0000 7ff7908b7700 1 -- 172.21.15.23:0/1842234674 >> 172.21.15.23:0/1842234674 conn(0x7ff78c0f1de0 msgr2=0x7ff78c0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:07.122+0000 7ff7908b7700 1 -- 172.21.15.23:0/1842234674 shutdown_connections 2021-11-19T08:44:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:07.122+0000 7ff7908b7700 1 -- 172.21.15.23:0/1842234674 wait complete. 2021-11-19T08:44:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 74 2021-11-19T08:44:07.120 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:43:52.800166+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 53, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24411, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24411': {'gid': 24411, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 65, 'state': 'up:clientreplay', 'state_seq': 17, 'addr': '172.21.15.134:6825/464801254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 464801254}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 464801254}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:07.121 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24411:up:clientreplay 2021-11-19T08:44:07.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:07 smithi023 conmon[33490]: audit 2021-11-19T08:44:07.121406+0000 mon.smithi023 (mon.0) 800 : audit 2021-11-19T08:44:07.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:07 smithi023 conmon[33490]: [DBG] from='client.? 172.21.15.23:0/1842234674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:07.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:07 smithi134 conmon[36363]: audit 2021-11-19T08:44:07.121406+0000 mon.smithi023 (mon.0) 800 : audit [DBG] from='client.? 172.21.15.23:0/1842234674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:08.122 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:08.261 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.275+0000 7f0d20cb5700 1 -- 172.21.15.23:0/4034198073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0f6840 msgr2=0x7f0d1c0f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:08.262 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.275+0000 7f0d20cb5700 1 --2- 172.21.15.23:0/4034198073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0f6840 0x7f0d1c0f6ca0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f0d10004660 tx=0x7f0d10009b30).stop 2021-11-19T08:44:08.262 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.275+0000 7f0d20cb5700 1 -- 172.21.15.23:0/4034198073 shutdown_connections 2021-11-19T08:44:08.262 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.275+0000 7f0d20cb5700 1 --2- 172.21.15.23:0/4034198073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0f6840 0x7f0d1c0f6ca0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:08.262 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.275+0000 7f0d20cb5700 1 --2- 172.21.15.23:0/4034198073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d1c0f55a0 0x7f0d1c0f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:08.263 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.275+0000 7f0d20cb5700 1 -- 172.21.15.23:0/4034198073 >> 172.21.15.23:0/4034198073 conn(0x7f0d1c0f0ce0 msgr2=0x7f0d1c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:08.263 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.275+0000 7f0d20cb5700 1 -- 172.21.15.23:0/4034198073 shutdown_connections 2021-11-19T08:44:08.263 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.275+0000 7f0d20cb5700 1 -- 172.21.15.23:0/4034198073 wait complete. 2021-11-19T08:44:08.263 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.275+0000 7f0d20cb5700 1 Processor -- start 2021-11-19T08:44:08.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.275+0000 7f0d20cb5700 1 -- start start 2021-11-19T08:44:08.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d20cb5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d1c0f55a0 0x7f0d1c05ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:08.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d20cb5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0f6840 0x7f0d1c10bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:08.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d20cb5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c10c1a0 con 0x7f0d1c0f6840 2021-11-19T08:44:08.264 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d20cb5700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c10c2e0 con 0x7f0d1c0f55a0 2021-11-19T08:44:08.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d19d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0f6840 0x7f0d1c10bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:08.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d19d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0f6840 0x7f0d1c10bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48260/0 (socket says 172.21.15.23:48260) 2021-11-19T08:44:08.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d19d9b700 1 -- 172.21.15.23:0/984642560 learned_addr learned my addr 172.21.15.23:0/984642560 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:08.265 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d1a59c700 1 --2- 172.21.15.23:0/984642560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d1c0f55a0 0x7f0d1c05ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:08.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d19d9b700 1 -- 172.21.15.23:0/984642560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d1c0f55a0 msgr2=0x7f0d1c05ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:08.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d19d9b700 1 --2- 172.21.15.23:0/984642560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d1c0f55a0 0x7f0d1c05ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:08.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d19d9b700 1 -- 172.21.15.23:0/984642560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d10005040 con 0x7f0d1c0f6840 2021-11-19T08:44:08.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d19d9b700 1 --2- 172.21.15.23:0/984642560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0f6840 0x7f0d1c10bcb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f0d10009830 tx=0x7f0d10009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:08.266 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d0affd700 1 -- 172.21.15.23:0/984642560 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d10018070 con 0x7f0d1c0f6840 2021-11-19T08:44:08.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d0affd700 1 -- 172.21.15.23:0/984642560 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d10013410 con 0x7f0d1c0f6840 2021-11-19T08:44:08.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d20cb5700 1 -- 172.21.15.23:0/984642560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1c10c4e0 con 0x7f0d1c0f6840 2021-11-19T08:44:08.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d0affd700 1 -- 172.21.15.23:0/984642560 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d1001c620 con 0x7f0d1c0f6840 2021-11-19T08:44:08.267 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.276+0000 7f0d20cb5700 1 -- 172.21.15.23:0/984642560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c10ca10 con 0x7f0d1c0f6840 2021-11-19T08:44:08.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.277+0000 7f0d20cb5700 1 -- 172.21.15.23:0/984642560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c046330 con 0x7f0d1c0f6840 2021-11-19T08:44:08.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.278+0000 7f0d0affd700 1 -- 172.21.15.23:0/984642560 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0d1001c8e0 con 0x7f0d1c0f6840 2021-11-19T08:44:08.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.278+0000 7f0d0affd700 1 --2- 172.21.15.23:0/984642560 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0d0405c2a0 0x7f0d0405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:08.270 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.278+0000 7f0d0affd700 1 -- 172.21.15.23:0/984642560 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0d10078220 con 0x7f0d1c0f6840 2021-11-19T08:44:08.270 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.278+0000 7f0d1a59c700 1 --2- 172.21.15.23:0/984642560 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0d0405c2a0 0x7f0d0405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:08.270 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.279+0000 7f0d1a59c700 1 --2- 172.21.15.23:0/984642560 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0d0405c2a0 0x7f0d0405e760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f0d1c0f66a0 tx=0x7f0d0c009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:08.270 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.282+0000 7f0d0affd700 1 -- 172.21.15.23:0/984642560 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d1004aa20 con 0x7f0d1c0f6840 2021-11-19T08:44:08.474 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.489+0000 7f0d20cb5700 1 -- 172.21.15.23:0/984642560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d1c027a00 con 0x7f0d1c0f6840 2021-11-19T08:44:08.474 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.490+0000 7f0d0affd700 1 -- 172.21.15.23:0/984642560 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f0d1004e040 con 0x7f0d1c0f6840 2021-11-19T08:44:08.475 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:08.475 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":76,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:08.249306+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:replay","state_seq":3,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:08.476 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.492+0000 7f0d20cb5700 1 -- 172.21.15.23:0/984642560 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0d0405c2a0 msgr2=0x7f0d0405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:08.477 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.492+0000 7f0d20cb5700 1 --2- 172.21.15.23:0/984642560 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0d0405c2a0 0x7f0d0405e760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f0d1c0f66a0 tx=0x7f0d0c009040).stop 2021-11-19T08:44:08.477 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.492+0000 7f0d20cb5700 1 -- 172.21.15.23:0/984642560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0f6840 msgr2=0x7f0d1c10bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:08.477 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.492+0000 7f0d20cb5700 1 --2- 172.21.15.23:0/984642560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0f6840 0x7f0d1c10bcb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f0d10009830 tx=0x7f0d10009b30).stop 2021-11-19T08:44:08.477 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.492+0000 7f0d20cb5700 1 -- 172.21.15.23:0/984642560 shutdown_connections 2021-11-19T08:44:08.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.492+0000 7f0d20cb5700 1 --2- 172.21.15.23:0/984642560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d1c0f55a0 0x7f0d1c05ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:08.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.492+0000 7f0d20cb5700 1 --2- 172.21.15.23:0/984642560 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0d0405c2a0 0x7f0d0405e760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:08.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.492+0000 7f0d20cb5700 1 --2- 172.21.15.23:0/984642560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0f6840 0x7f0d1c10bcb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:08.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.492+0000 7f0d20cb5700 1 -- 172.21.15.23:0/984642560 >> 172.21.15.23:0/984642560 conn(0x7f0d1c0f0ce0 msgr2=0x7f0d1c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:08.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.492+0000 7f0d20cb5700 1 -- 172.21.15.23:0/984642560 shutdown_connections 2021-11-19T08:44:08.479 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:08.492+0000 7f0d20cb5700 1 -- 172.21.15.23:0/984642560 wait complete. 2021-11-19T08:44:08.479 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 76 2021-11-19T08:44:08.482 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:08 smithi023 conmon[33490]: cluster 2021-11-19T08:44:06.884618+0000 2021-11-19T08:44:08.483 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:08 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 303 : cluster [DBG] pgmap v276: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-11-19T08:44:08.483 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:08 smithi023 conmon[33490]: cluster 2021-11-19T08:44:07.296880+0000 mon.smithi023 (mon.0) 801 2021-11-19T08:44:08.483 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:08 smithi023 conmon[33490]: : cluster [INF] daemon mds.cephfs.smithi134.jqsnzo restarted 2021-11-19T08:44:08.484 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:08 smithi023 conmon[33490]: cluster 2021-11 2021-11-19T08:44:08.484 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:08 smithi023 conmon[33490]: -19T08:44:07.300705+0000 mon.smithi023 (mon 2021-11-19T08:44:08.484 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:08 smithi023 conmon[33490]: .0) 802 : cluster [DBG] osdmap e54: 6 total, 6 up, 6 in 2021-11-19T08:44:08.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:08.249306+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:08.490 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:replay 2021-11-19T08:44:08.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:08 smithi134 conmon[36363]: cluster 2021-11-19T08:44:06.884618+0000 mgr.smithi023.ehlpcl (mgr.14164) 303 : cluster [DBG] pgmap v276: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2021-11-19T08:44:08.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:08 smithi134 conmon[36363]: cluster 2021-11-19T08:44:07.296880+0000 mon.smithi023 (mon.0) 801 : cluster [INF] daemon mds.cephfs.smithi134.jqsnzo restarted 2021-11-19T08:44:08.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:08 smithi134 conmon[36363]: cluster 2021-11-19T08:44:07.300705+0000 mon.smithi023 (mon.0) 802 : cluster [DBG] osdmap e54: 6 total, 6 up, 6 in 2021-11-19T08:44:09.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:09 smithi023 conmon[33490]: cluster 2021-11-19T08:44:08.249170+0000 mon.smithi023 2021-11-19T08:44:09.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:09 smithi023 conmon[33490]: (mon.0) 803 : cluster [DBG] mds.? [v2:172.21.15.134:6824/955716436,v1:172.21.15.134:6825/955716436] up:boot 2021-11-19T08:44:09.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:09 smithi023 conmon[33490]: cluster 2021-11-19T08:44:08.249256+0000 mon.smithi023 (mon.0) 804 : cluster [INF] Standby daemon mds.cephfs.smithi023.wazanl assigned to filesystem cephfs as rank 0 2021-11-19T08:44:09.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:09 smithi023 conmon[33490]: cluster 2021-11-19T08:44:08.249609+0000 mon.smithi023 (mon.0) 805 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi023.fuvzkr=up:active} 3 up:standby, 1 failed 2021-11-19T08:44:09.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:09 smithi023 conmon[33490]: audit 2021-11-19T08:44:08.249703+0000 mon.smithi023 (mon.0) 806 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.jqsnzo"}]: dispatch 2021-11-19T08:44:09.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:09 smithi023 conmon[33490]: cluster 2021-11-19T08:44:08.253723+0000 mon.smithi023 (mon.0) 807 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:replay,1=cephfs.smithi023.fuvzkr=up:active} 2 up:standby 2021-11-19T08:44:09.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:09 smithi023 conmon[33490]: audit 2021-11-19T08:44:08.490999+0000 mon.smithi023 (mon.0) 808 : audit [DBG] from='client.? 172.21.15.23:0/984642560' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:09.492 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:09.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:09 smithi134 conmon[36363]: cluster 2021-11-19T08:44:08.249170+0000 mon.smithi023 (mon.0) 803 : cluster [DBG] 2021-11-19T08:44:09.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:09 smithi134 conmon[36363]: mds.? [v2:172.21.15.134:6824/955716436,v1:172.21.15.134:6825/955716436] up:boot 2021-11-19T08:44:09.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:09 smithi134 conmon[36363]: cluster 2021-11-19T08:44:08.249256+0000 mon.smithi023 (mon.0) 804 : cluster [INF] Standby daemon mds.cephfs.smithi023.wazanl assigned to filesystem cephfs as rank 0 2021-11-19T08:44:09.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:09 smithi134 conmon[36363]: cluster 2021-11-19T08:44:08.249609+0000 mon.smithi023 (mon.0) 805 : cluster [DBG] fsmap cephfs:1/2 {1=cephfs.smithi023.fuvzkr=up:active} 3 up:standby, 1 failed 2021-11-19T08:44:09.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:09 smithi134 conmon[36363]: audit 2021-11-19T08:44:08.249703+0000 mon.smithi023 (mon.0) 806 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.jqsnzo"}]: dispatch 2021-11-19T08:44:09.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:09 smithi134 conmon[36363]: cluster 2021 2021-11-19T08:44:09.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:09 smithi134 conmon[36363]: -11-19T08:44:08.253723+0000 mon.smithi023 (mon.0) 807 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:replay,1=cephfs.smithi023.fuvzkr=up:active} 2 up:standby 2021-11-19T08:44:09.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:09 smithi134 conmon[36363]: audit 2021-11-19T08:44:08.490999+0000 mon.smithi023 (mon.0) 808 : 2021-11-19T08:44:09.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:09 smithi134 conmon[36363]: audit [DBG] from='client.? 172.21.15.23:0/984642560' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:09.631 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.643+0000 7f2893b13700 1 -- 172.21.15.23:0/722396005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f288c0f55a0 msgr2=0x7f288c0f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:09.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.643+0000 7f2893b13700 1 --2- 172.21.15.23:0/722396005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f288c0f55a0 0x7f288c0f59c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f2884004660 tx=0x7f2884009b30).stop 2021-11-19T08:44:09.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.644+0000 7f2893b13700 1 -- 172.21.15.23:0/722396005 shutdown_connections 2021-11-19T08:44:09.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.644+0000 7f2893b13700 1 --2- 172.21.15.23:0/722396005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f288c0f6840 0x7f288c0f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:09.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.644+0000 7f2893b13700 1 --2- 172.21.15.23:0/722396005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f288c0f55a0 0x7f288c0f59c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:09.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.644+0000 7f2893b13700 1 -- 172.21.15.23:0/722396005 >> 172.21.15.23:0/722396005 conn(0x7f288c0f0ce0 msgr2=0x7f288c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:09.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.644+0000 7f2893b13700 1 -- 172.21.15.23:0/722396005 shutdown_connections 2021-11-19T08:44:09.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.644+0000 7f2893b13700 1 -- 172.21.15.23:0/722396005 wait complete. 2021-11-19T08:44:09.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.644+0000 7f2893b13700 1 Processor -- start 2021-11-19T08:44:09.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.644+0000 7f2893b13700 1 -- start start 2021-11-19T08:44:09.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.644+0000 7f2893b13700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f288c0f55a0 0x7f288c0fd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:09.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.644+0000 7f2893b13700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f288c0f6840 0x7f288c0fbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.644+0000 7f2893b13700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288c0fdaf0 con 0x7f288c0f6840 2021-11-19T08:44:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.644+0000 7f2893b13700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288c0fc090 con 0x7f288c0f55a0 2021-11-19T08:44:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.645+0000 7f28910ae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f288c0f6840 0x7f288c0fbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:09.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.645+0000 7f28910ae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f288c0f6840 0x7f288c0fbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48278/0 (socket says 172.21.15.23:48278) 2021-11-19T08:44:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.645+0000 7f28910ae700 1 -- 172.21.15.23:0/791713541 learned_addr learned my addr 172.21.15.23:0/791713541 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.645+0000 7f28918af700 1 --2- 172.21.15.23:0/791713541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f288c0f55a0 0x7f288c0fd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.645+0000 7f28910ae700 1 -- 172.21.15.23:0/791713541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f288c0f55a0 msgr2=0x7f288c0fd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:09.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.645+0000 7f28910ae700 1 --2- 172.21.15.23:0/791713541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f288c0f55a0 0x7f288c0fd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.645+0000 7f28910ae700 1 -- 172.21.15.23:0/791713541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2884005040 con 0x7f288c0f6840 2021-11-19T08:44:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.645+0000 7f28910ae700 1 --2- 172.21.15.23:0/791713541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f288c0f6840 0x7f288c0fbba0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f28800026d0 tx=0x7f2880009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.645+0000 7f288a7fc700 1 -- 172.21.15.23:0/791713541 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28800079c0 con 0x7f288c0f6840 2021-11-19T08:44:09.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.645+0000 7f2893b13700 1 -- 172.21.15.23:0/791713541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f288c0fc310 con 0x7f288c0f6840 2021-11-19T08:44:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.645+0000 7f288a7fc700 1 -- 172.21.15.23:0/791713541 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f288000f040 con 0x7f288c0f6840 2021-11-19T08:44:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.645+0000 7f288a7fc700 1 -- 172.21.15.23:0/791713541 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28800148b0 con 0x7f288c0f6840 2021-11-19T08:44:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.645+0000 7f2893b13700 1 -- 172.21.15.23:0/791713541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288c0fc8d0 con 0x7f288c0f6840 2021-11-19T08:44:09.638 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.646+0000 7f2893b13700 1 -- 172.21.15.23:0/791713541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288c027a00 con 0x7f288c0f6840 2021-11-19T08:44:09.639 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.646+0000 7f288a7fc700 1 -- 172.21.15.23:0/791713541 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2880007b20 con 0x7f288c0f6840 2021-11-19T08:44:09.639 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.647+0000 7f288a7fc700 1 --2- 172.21.15.23:0/791713541 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f287405c2a0 0x7f287405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:09.639 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.647+0000 7f288a7fc700 1 -- 172.21.15.23:0/791713541 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f28800762a0 con 0x7f288c0f6840 2021-11-19T08:44:09.639 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.647+0000 7f28918af700 1 --2- 172.21.15.23:0/791713541 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f287405c2a0 0x7f287405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:09.640 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.647+0000 7f28918af700 1 --2- 172.21.15.23:0/791713541 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f287405c2a0 0x7f287405e760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f2884009f50 tx=0x7f2884009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:09.640 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.651+0000 7f288a7fc700 1 -- 172.21.15.23:0/791713541 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2880048b00 con 0x7f288c0f6840 2021-11-19T08:44:09.832 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.847+0000 7f2893b13700 1 -- 172.21.15.23:0/791713541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f288c0fee30 con 0x7f288c0f6840 2021-11-19T08:44:09.832 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.848+0000 7f288a7fc700 1 -- 172.21.15.23:0/791713541 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f288004c120 con 0x7f288c0f6840 2021-11-19T08:44:09.832 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:09.833 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":77,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:09.251877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:09.834 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.849+0000 7f2893b13700 1 -- 172.21.15.23:0/791713541 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f287405c2a0 msgr2=0x7f287405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:09.834 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.849+0000 7f2893b13700 1 --2- 172.21.15.23:0/791713541 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f287405c2a0 0x7f287405e760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f2884009f50 tx=0x7f2884009b30).stop 2021-11-19T08:44:09.835 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.849+0000 7f2893b13700 1 -- 172.21.15.23:0/791713541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f288c0f6840 msgr2=0x7f288c0fbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:09.835 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.849+0000 7f2893b13700 1 --2- 172.21.15.23:0/791713541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f288c0f6840 0x7f288c0fbba0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f28800026d0 tx=0x7f2880009de0).stop 2021-11-19T08:44:09.835 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.849+0000 7f2893b13700 1 -- 172.21.15.23:0/791713541 shutdown_connections 2021-11-19T08:44:09.835 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.849+0000 7f2893b13700 1 --2- 172.21.15.23:0/791713541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f288c0f55a0 0x7f288c0fd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:09.836 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.849+0000 7f2893b13700 1 --2- 172.21.15.23:0/791713541 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f287405c2a0 0x7f287405e760 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:09.836 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.849+0000 7f2893b13700 1 --2- 172.21.15.23:0/791713541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f288c0f6840 0x7f288c0fbba0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:09.836 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.849+0000 7f2893b13700 1 -- 172.21.15.23:0/791713541 >> 172.21.15.23:0/791713541 conn(0x7f288c0f0ce0 msgr2=0x7f288c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:09.836 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.849+0000 7f2893b13700 1 -- 172.21.15.23:0/791713541 shutdown_connections 2021-11-19T08:44:09.837 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:09.849+0000 7f2893b13700 1 -- 172.21.15.23:0/791713541 wait complete. 2021-11-19T08:44:09.837 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 77 2021-11-19T08:44:09.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:09.251877+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:09.847 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:resolve 2021-11-19T08:44:10.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:10 smithi134 conmon[36363]: cluster 2021-11-19T08:44:08.885068+0000 mgr.smithi023.ehlpcl (mgr.14164) 304 : cluster 2021-11-19T08:44:10.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:10 smithi134 conmon[36363]: [DBG] pgmap v278: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:44:10.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:10 smithi134 conmon[36363]: cluster 2021-11-19T08:44:09.257737+0000 mon.smithi023 (mon.0) 809 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3188238627,v1:172.21.15.23:6829/3188238627] up:resolve 2021-11-19T08:44:10.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:10 smithi134 conmon[36363]: cluster 2021-11-19T08:44:09.257792+0000 mon.smithi023 (mon.0 2021-11-19T08:44:10.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:10 smithi134 conmon[36363]: ) 810 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi023.fuvzkr=up:active} 2 up:standby 2021-11-19T08:44:10.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:10 smithi134 conmon[36363]: audit 2021-11-19T08:44:09.848714+0000 mon.smithi023 (mon.0) 811 : audit [DBG] from='client.? 172.21.15.23:0/791713541' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:10.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:10 smithi023 conmon[33490]: cluster 2021-11-19T08:44:08.885068+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:44:10.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:10 smithi023 conmon[33490]: .14164) 304 : cluster [DBG] pgmap v278: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:44:10.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:10 smithi023 conmon[33490]: cluster 2021-11-19T08:44:09.257737+0000 mon.smithi023 (mon.0) 809 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3188238627,v1:172.21.15.23:6829/3188238627] up:resolve 2021-11-19T08:44:10.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:10 smithi023 conmon[33490]: cluster 2021-11-19T08:44:09.257792+0000 mon.smithi023 (mon.0) 810 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi023.fuvzkr=up:active} 2 up:standby 2021-11-19T08:44:10.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:10 smithi023 conmon[33490]: audit 2021-11-19T08:44:09.848714+0000 2021-11-19T08:44:10.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:10 smithi023 conmon[33490]: mon.smithi023 (mon.0) 811 : audit [DBG] from='client.? 172.21.15.23:0/791713541' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:10.848 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:10.982 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.995+0000 7fda1f553700 1 -- 172.21.15.23:0/1179269695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda180f54d0 msgr2=0x7fda180f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:10.982 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.995+0000 7fda1f553700 1 --2- 172.21.15.23:0/1179269695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda180f54d0 0x7fda180f58f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fda14004660 tx=0x7fda14009b30).stop 2021-11-19T08:44:10.983 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1f553700 1 -- 172.21.15.23:0/1179269695 shutdown_connections 2021-11-19T08:44:10.983 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1f553700 1 --2- 172.21.15.23:0/1179269695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda180f6770 0x7fda180f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:10.983 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1f553700 1 --2- 172.21.15.23:0/1179269695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda180f54d0 0x7fda180f58f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:10.983 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1f553700 1 -- 172.21.15.23:0/1179269695 >> 172.21.15.23:0/1179269695 conn(0x7fda180f0c10 msgr2=0x7fda180f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:10.984 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1f553700 1 -- 172.21.15.23:0/1179269695 shutdown_connections 2021-11-19T08:44:10.984 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1f553700 1 -- 172.21.15.23:0/1179269695 wait complete. 2021-11-19T08:44:10.984 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1f553700 1 Processor -- start 2021-11-19T08:44:10.984 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1f553700 1 -- start start 2021-11-19T08:44:10.985 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1f553700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda180f54d0 0x7fda18106d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:10.985 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1f553700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda180f6770 0x7fda18107250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:10.985 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1f553700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda18107850 con 0x7fda180f6770 2021-11-19T08:44:10.985 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1f553700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda18108060 con 0x7fda180f54d0 2021-11-19T08:44:10.985 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1dd50700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda180f6770 0x7fda18107250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:10.986 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1dd50700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda180f6770 0x7fda18107250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48294/0 (socket says 172.21.15.23:48294) 2021-11-19T08:44:10.986 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.996+0000 7fda1dd50700 1 -- 172.21.15.23:0/1803179152 learned_addr learned my addr 172.21.15.23:0/1803179152 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:10.986 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.997+0000 7fda1e551700 1 --2- 172.21.15.23:0/1803179152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda180f54d0 0x7fda18106d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:10.986 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.997+0000 7fda1dd50700 1 -- 172.21.15.23:0/1803179152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda180f54d0 msgr2=0x7fda18106d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:10.987 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.997+0000 7fda1dd50700 1 --2- 172.21.15.23:0/1803179152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda180f54d0 0x7fda18106d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:10.987 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.997+0000 7fda1dd50700 1 -- 172.21.15.23:0/1803179152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda14005040 con 0x7fda180f6770 2021-11-19T08:44:10.987 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.997+0000 7fda1dd50700 1 --2- 172.21.15.23:0/1803179152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda180f6770 0x7fda18107250 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fda08002700 tx=0x7fda08009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:10.987 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.997+0000 7fda0effd700 1 -- 172.21.15.23:0/1803179152 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda08009320 con 0x7fda180f6770 2021-11-19T08:44:10.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.997+0000 7fda0effd700 1 -- 172.21.15.23:0/1803179152 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda0800f040 con 0x7fda180f6770 2021-11-19T08:44:10.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.997+0000 7fda1f553700 1 -- 172.21.15.23:0/1803179152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda181082e0 con 0x7fda180f6770 2021-11-19T08:44:10.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.997+0000 7fda0effd700 1 -- 172.21.15.23:0/1803179152 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda08014870 con 0x7fda180f6770 2021-11-19T08:44:10.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.997+0000 7fda1f553700 1 -- 172.21.15.23:0/1803179152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda181088a0 con 0x7fda180f6770 2021-11-19T08:44:10.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.998+0000 7fda1f553700 1 -- 172.21.15.23:0/1803179152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda180f48c0 con 0x7fda180f6770 2021-11-19T08:44:10.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.999+0000 7fda0effd700 1 -- 172.21.15.23:0/1803179152 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fda08009480 con 0x7fda180f6770 2021-11-19T08:44:10.991 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:10.999+0000 7fda0effd700 1 --2- 172.21.15.23:0/1803179152 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fda0405c300 0x7fda0405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:10.991 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.000+0000 7fda0effd700 1 -- 172.21.15.23:0/1803179152 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fda080768b0 con 0x7fda180f6770 2021-11-19T08:44:10.991 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.000+0000 7fda1e551700 1 --2- 172.21.15.23:0/1803179152 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fda0405c300 0x7fda0405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:10.991 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.000+0000 7fda1e551700 1 --2- 172.21.15.23:0/1803179152 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fda0405c300 0x7fda0405e7c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fda18106ad0 tx=0x7fda14009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:10.992 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.003+0000 7fda0effd700 1 -- 172.21.15.23:0/1803179152 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda08049130 con 0x7fda180f6770 2021-11-19T08:44:11.185 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.201+0000 7fda1f553700 1 -- 172.21.15.23:0/1803179152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda18045ad0 con 0x7fda180f6770 2021-11-19T08:44:11.186 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.202+0000 7fda0effd700 1 -- 172.21.15.23:0/1803179152 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2952 (secure 0 0 0) 0x7fda0804c750 con 0x7fda180f6770 2021-11-19T08:44:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:11.187 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":77,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:09.251877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:11.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.203+0000 7fda1f553700 1 -- 172.21.15.23:0/1803179152 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fda0405c300 msgr2=0x7fda0405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:11.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.203+0000 7fda1f553700 1 --2- 172.21.15.23:0/1803179152 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fda0405c300 0x7fda0405e7c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fda18106ad0 tx=0x7fda14009b30).stop 2021-11-19T08:44:11.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.203+0000 7fda1f553700 1 -- 172.21.15.23:0/1803179152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda180f6770 msgr2=0x7fda18107250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:11.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.203+0000 7fda1f553700 1 --2- 172.21.15.23:0/1803179152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda180f6770 0x7fda18107250 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fda08002700 tx=0x7fda08009e00).stop 2021-11-19T08:44:11.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.203+0000 7fda1f553700 1 -- 172.21.15.23:0/1803179152 shutdown_connections 2021-11-19T08:44:11.190 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.203+0000 7fda1f553700 1 --2- 172.21.15.23:0/1803179152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fda180f54d0 0x7fda18106d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:11.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.203+0000 7fda1f553700 1 --2- 172.21.15.23:0/1803179152 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fda0405c300 0x7fda0405e7c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:11.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.203+0000 7fda1f553700 1 --2- 172.21.15.23:0/1803179152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda180f6770 0x7fda18107250 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:11.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.203+0000 7fda1f553700 1 -- 172.21.15.23:0/1803179152 >> 172.21.15.23:0/1803179152 conn(0x7fda180f0c10 msgr2=0x7fda180fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:11.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.203+0000 7fda1f553700 1 -- 172.21.15.23:0/1803179152 shutdown_connections 2021-11-19T08:44:11.191 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:11.203+0000 7fda1f553700 1 -- 172.21.15.23:0/1803179152 wait complete. 2021-11-19T08:44:11.192 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 77 2021-11-19T08:44:11.200 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:09.251877+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:11.200 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:resolve 2021-11-19T08:44:11.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:11 smithi134 conmon[36363]: audit 2021-11-19T08:44:11.202864+0000 mon.smithi023 (mon.0) 2021-11-19T08:44:11.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:11 smithi134 conmon[36363]: 812 : audit [DBG] from='client.? 172.21.15.23:0/1803179152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:11.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:11 smithi023 conmon[33490]: audit 2021-11-19T08:44:11.202864+0000 mon.smithi023 (mon.0) 812 2021-11-19T08:44:11.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:11 smithi023 conmon[33490]: : audit [DBG] from='client.? 172.21.15.23:0/1803179152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:12.202 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:12.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.356+0000 7f2520bf2700 1 -- 172.21.15.23:0/1150720373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f251c060470 msgr2=0x7f251c0fb5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:12.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.356+0000 7f2520bf2700 1 --2- 172.21.15.23:0/1150720373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f251c060470 0x7f251c0fb5b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f2510004660 tx=0x7f2510009b30).stop 2021-11-19T08:44:12.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.356+0000 7f2520bf2700 1 -- 172.21.15.23:0/1150720373 shutdown_connections 2021-11-19T08:44:12.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.356+0000 7f2520bf2700 1 --2- 172.21.15.23:0/1150720373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f251c0fbaa0 0x7f251c0fde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:12.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.356+0000 7f2520bf2700 1 --2- 172.21.15.23:0/1150720373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f251c060470 0x7f251c0fb5b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:12.344 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.356+0000 7f2520bf2700 1 -- 172.21.15.23:0/1150720373 >> 172.21.15.23:0/1150720373 conn(0x7f251c0f0c30 msgr2=0x7f251c0f3050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:12.344 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.356+0000 7f2520bf2700 1 -- 172.21.15.23:0/1150720373 shutdown_connections 2021-11-19T08:44:12.344 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.356+0000 7f2520bf2700 1 -- 172.21.15.23:0/1150720373 wait complete. 2021-11-19T08:44:12.344 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.356+0000 7f2520bf2700 1 Processor -- start 2021-11-19T08:44:12.345 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.356+0000 7f2520bf2700 1 -- start start 2021-11-19T08:44:12.345 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f2520bf2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f251c060470 0x7f251c108f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:12.345 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f2520bf2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f251c0fbaa0 0x7f251c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:12.345 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f2520bf2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c109a00 con 0x7f251c0fbaa0 2021-11-19T08:44:12.345 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f2520bf2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c109b40 con 0x7f251c060470 2021-11-19T08:44:12.346 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f251affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f251c0fbaa0 0x7f251c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:12.346 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f251affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f251c0fbaa0 0x7f251c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48308/0 (socket says 172.21.15.23:48308) 2021-11-19T08:44:12.346 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f251affd700 1 -- 172.21.15.23:0/3572171294 learned_addr learned my addr 172.21.15.23:0/3572171294 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:12.346 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f251b7fe700 1 --2- 172.21.15.23:0/3572171294 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f251c060470 0x7f251c108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:12.346 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f251affd700 1 -- 172.21.15.23:0/3572171294 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f251c060470 msgr2=0x7f251c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:12.347 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f251affd700 1 --2- 172.21.15.23:0/3572171294 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f251c060470 0x7f251c108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:12.347 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f251affd700 1 -- 172.21.15.23:0/3572171294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2510005040 con 0x7f251c0fbaa0 2021-11-19T08:44:12.347 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f251affd700 1 --2- 172.21.15.23:0/3572171294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f251c0fbaa0 0x7f251c109450 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f250c00c790 tx=0x7f250c007d40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:12.347 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f2503fff700 1 -- 172.21.15.23:0/3572171294 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f250c013630 con 0x7f251c0fbaa0 2021-11-19T08:44:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f2503fff700 1 -- 172.21.15.23:0/3572171294 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f250c013c40 con 0x7f251c0fbaa0 2021-11-19T08:44:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.357+0000 7f2520bf2700 1 -- 172.21.15.23:0/3572171294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f251c10a4e0 con 0x7f251c0fbaa0 2021-11-19T08:44:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.358+0000 7f2503fff700 1 -- 172.21.15.23:0/3572171294 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f250c002b80 con 0x7f251c0fbaa0 2021-11-19T08:44:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.358+0000 7f2520bf2700 1 -- 172.21.15.23:0/3572171294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f251c10aaa0 con 0x7f251c0fbaa0 2021-11-19T08:44:12.349 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.359+0000 7f2520bf2700 1 -- 172.21.15.23:0/3572171294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251c045ad0 con 0x7f251c0fbaa0 2021-11-19T08:44:12.350 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.359+0000 7f2503fff700 1 -- 172.21.15.23:0/3572171294 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f250c01a060 con 0x7f251c0fbaa0 2021-11-19T08:44:12.350 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.359+0000 7f2503fff700 1 --2- 172.21.15.23:0/3572171294 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f250405c300 0x7f250405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:12.350 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.359+0000 7f2503fff700 1 -- 172.21.15.23:0/3572171294 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f250c0764a0 con 0x7f251c0fbaa0 2021-11-19T08:44:12.350 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.359+0000 7f251b7fe700 1 --2- 172.21.15.23:0/3572171294 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f250405c300 0x7f250405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.360+0000 7f251b7fe700 1 --2- 172.21.15.23:0/3572171294 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f250405c300 0x7f250405e7c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f2510004630 tx=0x7f2510009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.363+0000 7f2503fff700 1 -- 172.21.15.23:0/3572171294 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f250c048d00 con 0x7f251c0fbaa0 2021-11-19T08:44:12.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:12 smithi134 conmon[36363]: cluster 2021-11-19T08:44:10.885738+0000 mgr.smithi023.ehlpcl (mgr.14164) 305 : cluster [DBG] 2021-11-19T08:44:12.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:12 smithi134 conmon[36363]: pgmap v279: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:12.543 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.559+0000 7f2520bf2700 1 -- 172.21.15.23:0/3572171294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f251c027a00 con 0x7f251c0fbaa0 2021-11-19T08:44:12.544 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.560+0000 7f2503fff700 1 -- 172.21.15.23:0/3572171294 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f250c04c320 con 0x7f251c0fbaa0 2021-11-19T08:44:12.546 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:12.546 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:09.251877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:12.548 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.561+0000 7f2520bf2700 1 -- 172.21.15.23:0/3572171294 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f250405c300 msgr2=0x7f250405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:12.548 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.561+0000 7f2520bf2700 1 --2- 172.21.15.23:0/3572171294 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f250405c300 0x7f250405e7c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f2510004630 tx=0x7f2510009b30).stop 2021-11-19T08:44:12.548 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.561+0000 7f2520bf2700 1 -- 172.21.15.23:0/3572171294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f251c0fbaa0 msgr2=0x7f251c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:12.548 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.561+0000 7f2520bf2700 1 --2- 172.21.15.23:0/3572171294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f251c0fbaa0 0x7f251c109450 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f250c00c790 tx=0x7f250c007d40).stop 2021-11-19T08:44:12.549 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.561+0000 7f2520bf2700 1 -- 172.21.15.23:0/3572171294 shutdown_connections 2021-11-19T08:44:12.549 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.561+0000 7f2520bf2700 1 --2- 172.21.15.23:0/3572171294 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f251c060470 0x7f251c108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:12.549 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.561+0000 7f2520bf2700 1 --2- 172.21.15.23:0/3572171294 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f250405c300 0x7f250405e7c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:12.550 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.561+0000 7f2520bf2700 1 --2- 172.21.15.23:0/3572171294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f251c0fbaa0 0x7f251c109450 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:12.550 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.561+0000 7f2520bf2700 1 -- 172.21.15.23:0/3572171294 >> 172.21.15.23:0/3572171294 conn(0x7f251c0f0c30 msgr2=0x7f251c0f3020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:12.550 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.562+0000 7f2520bf2700 1 -- 172.21.15.23:0/3572171294 shutdown_connections 2021-11-19T08:44:12.550 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:12.562+0000 7f2520bf2700 1 -- 172.21.15.23:0/3572171294 wait complete. 2021-11-19T08:44:12.551 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 78 2021-11-19T08:44:12.556 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:12 smithi023 conmon[33490]: cluster 2021-11-19T08:44:10.885738+0000 mgr.smithi023.ehlpcl (mgr.14164) 305 : cluster [DBG] 2021-11-19T08:44:12.557 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:12 smithi023 conmon[33490]: pgmap v279: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:12.558 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:09.251877+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:12.558 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:resolve 2021-11-19T08:44:13.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:13 smithi134 conmon[36363]: cluster 2021-11-19T08:44:12.268604+0000 mon.smithi023 (mon.0) 813 : cluster [DBG] mds.? [v2:172.21.15.134:6824/955716436,v1:172.21.15.134:6825/955716436] up:standby 2021-11-19T08:44:13.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:13 smithi134 conmon[36363]: cluster 2021-11-19T08:44:12.268692+0000 mon.smithi023 (mon.0) 814 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi023.fuvzkr=up:active} 2 up:standby 2021-11-19T08:44:13.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:13 smithi134 conmon[36363]: audit 2021-11-19T08:44:12.560985+0000 mon.smithi023 (mon.0) 815 : audit [DBG] from='client.? 172.21.15.23:0/3572171294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:13.559 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:13.588 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:13 smithi023 conmon[33490]: cluster 2021-11-19T08:44:12.268604+0000 mon.smithi023 (mon.0) 813 : cluster [DBG] 2021-11-19T08:44:13.588 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:13 smithi023 conmon[33490]: mds.? [v2:172.21.15.134:6824/955716436,v1:172.21.15.134:6825/955716436] up:standby 2021-11-19T08:44:13.589 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:13 smithi023 conmon[33490]: cluster 2021-11-19T08:44:12 2021-11-19T08:44:13.589 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:13 smithi023 conmon[33490]: .268692+0000 mon.smithi023 (mon.0) 814 2021-11-19T08:44:13.589 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:13 smithi023 conmon[33490]: : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi023.fuvzkr=up:active} 2 up:standby 2021-11-19T08:44:13.589 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:13 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:44:13.590 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:13 smithi023 conmon[33490]: -19T08:44:12.560985+0000 mon.smithi023 ( 2021-11-19T08:44:13.590 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:13 smithi023 conmon[33490]: mon.0) 815 : audit [DBG] 2021-11-19T08:44:13.590 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:13 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/3572171294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:13.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.708+0000 7f946714a700 1 -- 172.21.15.23:0/2097425282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94600f7440 msgr2=0x7f94600fde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:13.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.708+0000 7f946714a700 1 --2- 172.21.15.23:0/2097425282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94600f7440 0x7f94600fde60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f9450004690 tx=0x7f9450009b30).stop 2021-11-19T08:44:13.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.708+0000 7f946714a700 1 -- 172.21.15.23:0/2097425282 shutdown_connections 2021-11-19T08:44:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.708+0000 7f946714a700 1 --2- 172.21.15.23:0/2097425282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94600f7440 0x7f94600fde60 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.708+0000 7f946714a700 1 --2- 172.21.15.23:0/2097425282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94600f6b30 0x7f94600f6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.708+0000 7f946714a700 1 -- 172.21.15.23:0/2097425282 >> 172.21.15.23:0/2097425282 conn(0x7f94600f0360 msgr2=0x7f94600f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.708+0000 7f946714a700 1 -- 172.21.15.23:0/2097425282 shutdown_connections 2021-11-19T08:44:13.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.708+0000 7f946714a700 1 -- 172.21.15.23:0/2097425282 wait complete. 2021-11-19T08:44:13.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.708+0000 7f946714a700 1 Processor -- start 2021-11-19T08:44:13.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.708+0000 7f946714a700 1 -- start start 2021-11-19T08:44:13.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f946714a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94600f6b30 0x7f9460108e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:13.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f946714a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94600f7440 0x7f9460109370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:13.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f946714a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9460109920 con 0x7f94600f6b30 2021-11-19T08:44:13.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f946714a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9460109a60 con 0x7f94600f7440 2021-11-19T08:44:13.698 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f9466148700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94600f6b30 0x7f9460108e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:13.699 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f9466148700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94600f6b30 0x7f9460108e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48316/0 (socket says 172.21.15.23:48316) 2021-11-19T08:44:13.699 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f9466148700 1 -- 172.21.15.23:0/1018782958 learned_addr learned my addr 172.21.15.23:0/1018782958 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:13.699 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f9465947700 1 --2- 172.21.15.23:0/1018782958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94600f7440 0x7f9460109370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:13.699 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f9466148700 1 -- 172.21.15.23:0/1018782958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94600f7440 msgr2=0x7f9460109370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:13.700 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f9466148700 1 --2- 172.21.15.23:0/1018782958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94600f7440 0x7f9460109370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:13.700 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f9466148700 1 -- 172.21.15.23:0/1018782958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9450005040 con 0x7f94600f6b30 2021-11-19T08:44:13.700 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f9466148700 1 --2- 172.21.15.23:0/1018782958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94600f6b30 0x7f9460108e80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f945c002700 tx=0x7f945c009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:13.700 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f9456ffd700 1 -- 172.21.15.23:0/1018782958 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f945c009320 con 0x7f94600f6b30 2021-11-19T08:44:13.701 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f9456ffd700 1 -- 172.21.15.23:0/1018782958 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f945c00f040 con 0x7f94600f6b30 2021-11-19T08:44:13.701 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f946714a700 1 -- 172.21.15.23:0/1018782958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946010a400 con 0x7f94600f6b30 2021-11-19T08:44:13.701 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f9456ffd700 1 -- 172.21.15.23:0/1018782958 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f945c014870 con 0x7f94600f6b30 2021-11-19T08:44:13.702 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.709+0000 7f946714a700 1 -- 172.21.15.23:0/1018782958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946010a940 con 0x7f94600f6b30 2021-11-19T08:44:13.702 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.710+0000 7f946714a700 1 -- 172.21.15.23:0/1018782958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9460045ad0 con 0x7f94600f6b30 2021-11-19T08:44:13.702 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.711+0000 7f9456ffd700 1 -- 172.21.15.23:0/1018782958 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f945c009480 con 0x7f94600f6b30 2021-11-19T08:44:13.703 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.711+0000 7f9456ffd700 1 --2- 172.21.15.23:0/1018782958 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f944c05c300 0x7f944c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:13.704 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.711+0000 7f9456ffd700 1 -- 172.21.15.23:0/1018782958 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f945c0765b0 con 0x7f94600f6b30 2021-11-19T08:44:13.704 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.711+0000 7f9465947700 1 --2- 172.21.15.23:0/1018782958 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f944c05c300 0x7f944c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:13.704 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.712+0000 7f9465947700 1 --2- 172.21.15.23:0/1018782958 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f944c05c300 0x7f944c05e7c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f9450009d60 tx=0x7f9450009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:13.704 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.715+0000 7f9456ffd700 1 -- 172.21.15.23:0/1018782958 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f945c048e10 con 0x7f94600f6b30 2021-11-19T08:44:13.895 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.910+0000 7f946714a700 1 -- 172.21.15.23:0/1018782958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9460027a00 con 0x7f94600f6b30 2021-11-19T08:44:13.896 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.911+0000 7f9456ffd700 1 -- 172.21.15.23:0/1018782958 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f945c04c430 con 0x7f94600f6b30 2021-11-19T08:44:13.896 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:13.897 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:09.251877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.913+0000 7f946714a700 1 -- 172.21.15.23:0/1018782958 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f944c05c300 msgr2=0x7f944c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.913+0000 7f946714a700 1 --2- 172.21.15.23:0/1018782958 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f944c05c300 0x7f944c05e7c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f9450009d60 tx=0x7f9450009b30).stop 2021-11-19T08:44:13.899 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.913+0000 7f946714a700 1 -- 172.21.15.23:0/1018782958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94600f6b30 msgr2=0x7f9460108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:13.899 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.913+0000 7f946714a700 1 --2- 172.21.15.23:0/1018782958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94600f6b30 0x7f9460108e80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f945c002700 tx=0x7f945c009e00).stop 2021-11-19T08:44:13.900 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.913+0000 7f946714a700 1 -- 172.21.15.23:0/1018782958 shutdown_connections 2021-11-19T08:44:13.900 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.913+0000 7f946714a700 1 --2- 172.21.15.23:0/1018782958 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f944c05c300 0x7f944c05e7c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:13.900 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.913+0000 7f946714a700 1 --2- 172.21.15.23:0/1018782958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94600f6b30 0x7f9460108e80 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:13.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.913+0000 7f946714a700 1 --2- 172.21.15.23:0/1018782958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94600f7440 0x7f9460109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:13.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.913+0000 7f946714a700 1 -- 172.21.15.23:0/1018782958 >> 172.21.15.23:0/1018782958 conn(0x7f94600f0360 msgr2=0x7f94600f2750 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:13.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.913+0000 7f946714a700 1 -- 172.21.15.23:0/1018782958 shutdown_connections 2021-11-19T08:44:13.901 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:13.913+0000 7f946714a700 1 -- 172.21.15.23:0/1018782958 wait complete. 2021-11-19T08:44:13.902 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 78 2021-11-19T08:44:13.909 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:09.251877+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:13.910 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:resolve 2021-11-19T08:44:14.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:14 smithi134 conmon[36363]: cluster 2021-11-19T08:44:12.886163+0000 mgr.smithi023.ehlpcl (mgr.14164) 306 : cluster [DBG] 2021-11-19T08:44:14.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:14 smithi134 conmon[36363]: pgmap v280: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:14.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:14 smithi134 conmon[36363]: audit 2021-11-19T08:44:13.912000+0000 mon.smithi023 (mon.0) 816 : audit [DBG] from='client.? 172.21.15.23:0/1018782958' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:14.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:14 smithi023 conmon[33490]: cluster 2021-11-19T08:44:12.886163+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:44:14.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:14 smithi023 conmon[33490]: .14164) 306 : cluster [DBG] pgmap v280: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:14.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:14 smithi023 conmon[33490]: audit 2021-11-19T08:44:13.912000+0000 mon.smithi023 (mon.0) 816 : 2021-11-19T08:44:14.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:14 smithi023 conmon[33490]: audit [DBG] from='client.? 172.21.15.23:0/1018782958' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:14.911 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:15.046 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.059+0000 7f5df4890700 1 -- 172.21.15.23:0/2237052771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df00f58b0 msgr2=0x7f5df00f5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:15.046 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.059+0000 7f5df4890700 1 --2- 172.21.15.23:0/2237052771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df00f58b0 0x7f5df00f5d30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f5dd8004660 tx=0x7f5dd8009b30).stop 2021-11-19T08:44:15.046 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.059+0000 7f5df4890700 1 -- 172.21.15.23:0/2237052771 shutdown_connections 2021-11-19T08:44:15.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.059+0000 7f5df4890700 1 --2- 172.21.15.23:0/2237052771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df00f58b0 0x7f5df00f5d30 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:15.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.059+0000 7f5df4890700 1 --2- 172.21.15.23:0/2237052771 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5df00f6bb0 0x7f5df00f6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:15.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.059+0000 7f5df4890700 1 -- 172.21.15.23:0/2237052771 >> 172.21.15.23:0/2237052771 conn(0x7f5df00f04c0 msgr2=0x7f5df00f28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:15.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.059+0000 7f5df4890700 1 -- 172.21.15.23:0/2237052771 shutdown_connections 2021-11-19T08:44:15.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.060+0000 7f5df4890700 1 -- 172.21.15.23:0/2237052771 wait complete. 2021-11-19T08:44:15.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.060+0000 7f5df4890700 1 Processor -- start 2021-11-19T08:44:15.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.060+0000 7f5df4890700 1 -- start start 2021-11-19T08:44:15.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.060+0000 7f5df4890700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5df00f58b0 0x7f5df0106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:15.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.060+0000 7f5df4890700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df00f6bb0 0x7f5df01073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.060+0000 7f5df4890700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df0107950 con 0x7f5df00f6bb0 2021-11-19T08:44:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.060+0000 7f5df4890700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df0107a90 con 0x7f5df00f58b0 2021-11-19T08:44:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.060+0000 7f5dedd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df00f6bb0 0x7f5df01073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.060+0000 7f5dedd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df00f6bb0 0x7f5df01073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48328/0 (socket says 172.21.15.23:48328) 2021-11-19T08:44:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.060+0000 7f5dedd9b700 1 -- 172.21.15.23:0/1046014249 learned_addr learned my addr 172.21.15.23:0/1046014249 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.060+0000 7f5dee59c700 1 --2- 172.21.15.23:0/1046014249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5df00f58b0 0x7f5df0106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.061+0000 7f5dedd9b700 1 -- 172.21.15.23:0/1046014249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5df00f58b0 msgr2=0x7f5df0106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.061+0000 7f5dedd9b700 1 --2- 172.21.15.23:0/1046014249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5df00f58b0 0x7f5df0106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.061+0000 7f5dedd9b700 1 -- 172.21.15.23:0/1046014249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd8005040 con 0x7f5df00f6bb0 2021-11-19T08:44:15.051 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.061+0000 7f5dedd9b700 1 --2- 172.21.15.23:0/1046014249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df00f6bb0 0x7f5df01073a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f5dd8002040 tx=0x7f5dd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:15.051 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.061+0000 7f5de6ffd700 1 -- 172.21.15.23:0/1046014249 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dd8019070 con 0x7f5df00f6bb0 2021-11-19T08:44:15.051 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.061+0000 7f5df4890700 1 -- 172.21.15.23:0/1046014249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df010c4d0 con 0x7f5df00f6bb0 2021-11-19T08:44:15.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.061+0000 7f5de6ffd700 1 -- 172.21.15.23:0/1046014249 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5dd8007e70 con 0x7f5df00f6bb0 2021-11-19T08:44:15.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.061+0000 7f5de6ffd700 1 -- 172.21.15.23:0/1046014249 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dd8014c20 con 0x7f5df00f6bb0 2021-11-19T08:44:15.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.061+0000 7f5df4890700 1 -- 172.21.15.23:0/1046014249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df010ca90 con 0x7f5df00f6bb0 2021-11-19T08:44:15.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.062+0000 7f5df4890700 1 -- 172.21.15.23:0/1046014249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df0027a00 con 0x7f5df00f6bb0 2021-11-19T08:44:15.053 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.062+0000 7f5de6ffd700 1 -- 172.21.15.23:0/1046014249 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5dd8025070 con 0x7f5df00f6bb0 2021-11-19T08:44:15.054 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.063+0000 7f5de6ffd700 1 --2- 172.21.15.23:0/1046014249 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5ddc05c2a0 0x7f5ddc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:15.054 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.063+0000 7f5de6ffd700 1 -- 172.21.15.23:0/1046014249 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f5dd8077290 con 0x7f5df00f6bb0 2021-11-19T08:44:15.054 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.063+0000 7f5dee59c700 1 --2- 172.21.15.23:0/1046014249 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5ddc05c2a0 0x7f5ddc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:15.054 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.063+0000 7f5dee59c700 1 --2- 172.21.15.23:0/1046014249 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5ddc05c2a0 0x7f5ddc05e760 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f5de8009fd0 tx=0x7f5de800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:15.055 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.067+0000 7f5de6ffd700 1 -- 172.21.15.23:0/1046014249 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5dd8049af0 con 0x7f5df00f6bb0 2021-11-19T08:44:15.251 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.267+0000 7f5df4890700 1 -- 172.21.15.23:0/1046014249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5df00feee0 con 0x7f5df00f6bb0 2021-11-19T08:44:15.252 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.268+0000 7f5de6ffd700 1 -- 172.21.15.23:0/1046014249 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f5dd8004490 con 0x7f5df00f6bb0 2021-11-19T08:44:15.253 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:15.253 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:09.251877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:15.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.269+0000 7f5df4890700 1 -- 172.21.15.23:0/1046014249 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5ddc05c2a0 msgr2=0x7f5ddc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:15.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.269+0000 7f5df4890700 1 --2- 172.21.15.23:0/1046014249 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5ddc05c2a0 0x7f5ddc05e760 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f5de8009fd0 tx=0x7f5de800b040).stop 2021-11-19T08:44:15.255 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.269+0000 7f5df4890700 1 -- 172.21.15.23:0/1046014249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df00f6bb0 msgr2=0x7f5df01073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:15.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.269+0000 7f5df4890700 1 --2- 172.21.15.23:0/1046014249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df00f6bb0 0x7f5df01073a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f5dd8002040 tx=0x7f5dd800b040).stop 2021-11-19T08:44:15.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.269+0000 7f5df4890700 1 -- 172.21.15.23:0/1046014249 shutdown_connections 2021-11-19T08:44:15.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.269+0000 7f5df4890700 1 --2- 172.21.15.23:0/1046014249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5df00f58b0 0x7f5df0106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:15.256 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.269+0000 7f5df4890700 1 --2- 172.21.15.23:0/1046014249 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f5ddc05c2a0 0x7f5ddc05e760 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:15.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.269+0000 7f5df4890700 1 --2- 172.21.15.23:0/1046014249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df00f6bb0 0x7f5df01073a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:15.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.270+0000 7f5df4890700 1 -- 172.21.15.23:0/1046014249 >> 172.21.15.23:0/1046014249 conn(0x7f5df00f04c0 msgr2=0x7f5df00fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:15.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.270+0000 7f5df4890700 1 -- 172.21.15.23:0/1046014249 shutdown_connections 2021-11-19T08:44:15.257 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:15.270+0000 7f5df4890700 1 -- 172.21.15.23:0/1046014249 wait complete. 2021-11-19T08:44:15.257 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 78 2021-11-19T08:44:15.267 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:09.251877+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:15.267 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:resolve 2021-11-19T08:44:15.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:15 smithi134 conmon[36363]: audit 2021-11-19T08:44:15.268908+0000 mon.smithi023 (mon.0) 817 : audit 2021-11-19T08:44:15.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:15 smithi134 conmon[36363]: [DBG] from='client.? 172.21.15.23:0/1046014249' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:15.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:15 smithi023 conmon[33490]: audit 2021-11-19T08:44:15.268908+0000 mon.smithi023 (mon.0) 817 : audit [DBG] from='client.? 172.21.15.23:0/1046014249' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:16.269 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:16.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:16 smithi023 conmon[33490]: cluster 2021-11-19T08:44:14.886859+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:44:16.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:16 smithi023 conmon[33490]: ) 307 : cluster [DBG] pgmap v281: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:16.417 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.430+0000 7f6deffb2700 1 -- 172.21.15.23:0/3983431746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de80f66a0 msgr2=0x7f6de80f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:16.417 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.430+0000 7f6deffb2700 1 --2- 172.21.15.23:0/3983431746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de80f66a0 0x7f6de80f6ac0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f6dd8004660 tx=0x7f6dd8009b30).stop 2021-11-19T08:44:16.417 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.430+0000 7f6deffb2700 1 -- 172.21.15.23:0/3983431746 shutdown_connections 2021-11-19T08:44:16.418 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.430+0000 7f6deffb2700 1 --2- 172.21.15.23:0/3983431746 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6de80f7940 0x7f6de80f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:16.418 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.430+0000 7f6deffb2700 1 --2- 172.21.15.23:0/3983431746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de80f66a0 0x7f6de80f6ac0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:16.418 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.430+0000 7f6deffb2700 1 -- 172.21.15.23:0/3983431746 >> 172.21.15.23:0/3983431746 conn(0x7f6de80f1e00 msgr2=0x7f6de80f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:16.418 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.430+0000 7f6deffb2700 1 -- 172.21.15.23:0/3983431746 shutdown_connections 2021-11-19T08:44:16.419 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.431+0000 7f6deffb2700 1 -- 172.21.15.23:0/3983431746 wait complete. 2021-11-19T08:44:16.419 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.431+0000 7f6deffb2700 1 Processor -- start 2021-11-19T08:44:16.419 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.431+0000 7f6deffb2700 1 -- start start 2021-11-19T08:44:16.420 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.431+0000 7f6deffb2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6de80f66a0 0x7f6de8106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:16.420 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.431+0000 7f6deffb2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de80f7940 0x7f6de8107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:16.420 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.431+0000 7f6deffb2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de8107930 con 0x7f6de80f7940 2021-11-19T08:44:16.420 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.431+0000 7f6deffb2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de8107a70 con 0x7f6de80f66a0 2021-11-19T08:44:16.421 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.431+0000 7f6ded54d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de80f7940 0x7f6de8107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:16.421 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.431+0000 7f6ded54d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de80f7940 0x7f6de8107380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48342/0 (socket says 172.21.15.23:48342) 2021-11-19T08:44:16.421 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.431+0000 7f6ded54d700 1 -- 172.21.15.23:0/1313052677 learned_addr learned my addr 172.21.15.23:0/1313052677 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:16.422 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.431+0000 7f6dedd4e700 1 --2- 172.21.15.23:0/1313052677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6de80f66a0 0x7f6de8106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:16.422 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.432+0000 7f6ded54d700 1 -- 172.21.15.23:0/1313052677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6de80f66a0 msgr2=0x7f6de8106e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:16.423 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.432+0000 7f6ded54d700 1 --2- 172.21.15.23:0/1313052677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6de80f66a0 0x7f6de8106e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:16.423 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.432+0000 7f6ded54d700 1 -- 172.21.15.23:0/1313052677 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd8005040 con 0x7f6de80f7940 2021-11-19T08:44:16.423 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.432+0000 7f6ded54d700 1 --2- 172.21.15.23:0/1313052677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de80f7940 0x7f6de8107380 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f6de40026d0 tx=0x7f6de4009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:16.423 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.432+0000 7f6dde7fc700 1 -- 172.21.15.23:0/1313052677 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6de40079c0 con 0x7f6de80f7940 2021-11-19T08:44:16.423 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.432+0000 7f6dde7fc700 1 -- 172.21.15.23:0/1313052677 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6de400f040 con 0x7f6de80f7940 2021-11-19T08:44:16.424 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.432+0000 7f6dde7fc700 1 -- 172.21.15.23:0/1313052677 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6de40148b0 con 0x7f6de80f7940 2021-11-19T08:44:16.424 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.432+0000 7f6deffb2700 1 -- 172.21.15.23:0/1313052677 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de810c510 con 0x7f6de80f7940 2021-11-19T08:44:16.424 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.432+0000 7f6deffb2700 1 -- 172.21.15.23:0/1313052677 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de810cad0 con 0x7f6de80f7940 2021-11-19T08:44:16.426 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.433+0000 7f6deffb2700 1 -- 172.21.15.23:0/1313052677 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de8027a00 con 0x7f6de80f7940 2021-11-19T08:44:16.426 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.433+0000 7f6dde7fc700 1 -- 172.21.15.23:0/1313052677 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6de4007b20 con 0x7f6de80f7940 2021-11-19T08:44:16.426 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.433+0000 7f6dde7fc700 1 --2- 172.21.15.23:0/1313052677 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6dd405c2a0 0x7f6dd405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:16.426 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.434+0000 7f6dde7fc700 1 -- 172.21.15.23:0/1313052677 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6de4076240 con 0x7f6de80f7940 2021-11-19T08:44:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.434+0000 7f6dedd4e700 1 --2- 172.21.15.23:0/1313052677 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6dd405c2a0 0x7f6dd405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.434+0000 7f6dedd4e700 1 --2- 172.21.15.23:0/1313052677 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6dd405c2a0 0x7f6dd405e760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f6dd8009f50 tx=0x7f6dd8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.438+0000 7f6dde7fc700 1 -- 172.21.15.23:0/1313052677 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6de4048b20 con 0x7f6de80f7940 2021-11-19T08:44:16.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:16 smithi134 conmon[36363]: cluster 2021-11-19T08:44:14.886859+0000 mgr.smithi023.ehlpcl (mgr.14164) 307 : cluster 2021-11-19T08:44:16.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:16 smithi134 conmon[36363]: [DBG] pgmap v281: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:16.618 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.633+0000 7f6deffb2700 1 -- 172.21.15.23:0/1313052677 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6de80fee30 con 0x7f6de80f7940 2021-11-19T08:44:16.618 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.634+0000 7f6dde7fc700 1 -- 172.21.15.23:0/1313052677 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f6de404c140 con 0x7f6de80f7940 2021-11-19T08:44:16.620 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:16.620 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:09.251877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:16.621 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.636+0000 7f6deffb2700 1 -- 172.21.15.23:0/1313052677 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6dd405c2a0 msgr2=0x7f6dd405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:16.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.636+0000 7f6deffb2700 1 --2- 172.21.15.23:0/1313052677 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6dd405c2a0 0x7f6dd405e760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f6dd8009f50 tx=0x7f6dd8009b30).stop 2021-11-19T08:44:16.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.636+0000 7f6deffb2700 1 -- 172.21.15.23:0/1313052677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de80f7940 msgr2=0x7f6de8107380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:16.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.636+0000 7f6deffb2700 1 --2- 172.21.15.23:0/1313052677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de80f7940 0x7f6de8107380 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f6de40026d0 tx=0x7f6de4009de0).stop 2021-11-19T08:44:16.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.636+0000 7f6deffb2700 1 -- 172.21.15.23:0/1313052677 shutdown_connections 2021-11-19T08:44:16.623 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.636+0000 7f6deffb2700 1 --2- 172.21.15.23:0/1313052677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6de80f66a0 0x7f6de8106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:16.623 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.636+0000 7f6deffb2700 1 --2- 172.21.15.23:0/1313052677 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f6dd405c2a0 0x7f6dd405e760 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:16.623 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.636+0000 7f6deffb2700 1 --2- 172.21.15.23:0/1313052677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de80f7940 0x7f6de8107380 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:16.623 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.636+0000 7f6deffb2700 1 -- 172.21.15.23:0/1313052677 >> 172.21.15.23:0/1313052677 conn(0x7f6de80f1e00 msgr2=0x7f6de80fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:16.624 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.636+0000 7f6deffb2700 1 -- 172.21.15.23:0/1313052677 shutdown_connections 2021-11-19T08:44:16.624 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:16.636+0000 7f6deffb2700 1 -- 172.21.15.23:0/1313052677 wait complete. 2021-11-19T08:44:16.625 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 78 2021-11-19T08:44:16.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:09.251877+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:16.632 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:resolve 2021-11-19T08:44:17.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:17 smithi134 conmon[36363]: audit 2021-11-19T08:44:16.634912+0000 mon.smithi023 (mon.0) 818 : audit 2021-11-19T08:44:17.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:17 smithi134 conmon[36363]: [DBG] from='client.? 172.21.15.23:0/1313052677' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:17.634 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:17.640 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:17 smithi023 conmon[33490]: audit 2021-11-19T08:44:16.634912 2021-11-19T08:44:17.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:17 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 818 : audit [DBG] from='client.? 172.21.15.23:0/1313052677' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.781+0000 7f4c3db92700 1 -- 172.21.15.23:0/372849121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f6610 msgr2=0x7f4c380f6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.781+0000 7f4c3db92700 1 --2- 172.21.15.23:0/372849121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f6610 0x7f4c380f6a70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f4c2c004660 tx=0x7f4c2c009b30).stop 2021-11-19T08:44:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.781+0000 7f4c3db92700 1 -- 172.21.15.23:0/372849121 shutdown_connections 2021-11-19T08:44:17.769 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.781+0000 7f4c3db92700 1 --2- 172.21.15.23:0/372849121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f6610 0x7f4c380f6a70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:17.770 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.781+0000 7f4c3db92700 1 --2- 172.21.15.23:0/372849121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c380f5370 0x7f4c380f5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:17.770 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.781+0000 7f4c3db92700 1 -- 172.21.15.23:0/372849121 >> 172.21.15.23:0/372849121 conn(0x7f4c380f0ab0 msgr2=0x7f4c380f2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:17.770 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.781+0000 7f4c3db92700 1 -- 172.21.15.23:0/372849121 shutdown_connections 2021-11-19T08:44:17.770 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.781+0000 7f4c3db92700 1 -- 172.21.15.23:0/372849121 wait complete. 2021-11-19T08:44:17.771 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.782+0000 7f4c3db92700 1 Processor -- start 2021-11-19T08:44:17.771 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.782+0000 7f4c3db92700 1 -- start start 2021-11-19T08:44:17.771 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.782+0000 7f4c3db92700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c380f5370 0x7f4c38109a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:17.772 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.782+0000 7f4c3db92700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f6610 0x7f4c3810bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:17.772 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.782+0000 7f4c3db92700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c38060450 con 0x7f4c380f6610 2021-11-19T08:44:17.772 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.782+0000 7f4c3db92700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c38060590 con 0x7f4c380f5370 2021-11-19T08:44:17.772 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.782+0000 7f4c37fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f6610 0x7f4c3810bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:17.772 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.782+0000 7f4c37fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f6610 0x7f4c3810bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48352/0 (socket says 172.21.15.23:48352) 2021-11-19T08:44:17.773 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.782+0000 7f4c37fff700 1 -- 172.21.15.23:0/1486336327 learned_addr learned my addr 172.21.15.23:0/1486336327 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:17.773 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.782+0000 7f4c3cb90700 1 --2- 172.21.15.23:0/1486336327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c380f5370 0x7f4c38109a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:17.773 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.782+0000 7f4c37fff700 1 -- 172.21.15.23:0/1486336327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c380f5370 msgr2=0x7f4c38109a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:17.773 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.782+0000 7f4c37fff700 1 --2- 172.21.15.23:0/1486336327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c380f5370 0x7f4c38109a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:17.774 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.782+0000 7f4c37fff700 1 -- 172.21.15.23:0/1486336327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c005040 con 0x7f4c380f6610 2021-11-19T08:44:17.774 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.783+0000 7f4c37fff700 1 --2- 172.21.15.23:0/1486336327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f6610 0x7f4c3810bf40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f4c2c009f50 tx=0x7f4c2c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:17.774 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.783+0000 7f4c357fa700 1 -- 172.21.15.23:0/1486336327 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c2c018070 con 0x7f4c380f6610 2021-11-19T08:44:17.774 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.783+0000 7f4c357fa700 1 -- 172.21.15.23:0/1486336327 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c2c013410 con 0x7f4c380f6610 2021-11-19T08:44:17.775 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.783+0000 7f4c357fa700 1 -- 172.21.15.23:0/1486336327 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c2c01c3c0 con 0x7f4c380f6610 2021-11-19T08:44:17.775 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.783+0000 7f4c3db92700 1 -- 172.21.15.23:0/1486336327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3810c490 con 0x7f4c380f6610 2021-11-19T08:44:17.775 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.783+0000 7f4c3db92700 1 -- 172.21.15.23:0/1486336327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3810c9f0 con 0x7f4c380f6610 2021-11-19T08:44:17.776 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.784+0000 7f4c3db92700 1 -- 172.21.15.23:0/1486336327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c38045ad0 con 0x7f4c380f6610 2021-11-19T08:44:17.776 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.784+0000 7f4c357fa700 1 -- 172.21.15.23:0/1486336327 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4c2c01d420 con 0x7f4c380f6610 2021-11-19T08:44:17.777 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.785+0000 7f4c357fa700 1 --2- 172.21.15.23:0/1486336327 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c2005c2b0 0x7f4c2005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:17.777 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.785+0000 7f4c357fa700 1 -- 172.21.15.23:0/1486336327 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4c2c077fe0 con 0x7f4c380f6610 2021-11-19T08:44:17.777 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.785+0000 7f4c3cb90700 1 --2- 172.21.15.23:0/1486336327 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c2005c2b0 0x7f4c2005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:17.777 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.785+0000 7f4c3cb90700 1 --2- 172.21.15.23:0/1486336327 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c2005c2b0 0x7f4c2005e770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f4c380600d0 tx=0x7f4c3000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:17.778 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.789+0000 7f4c357fa700 1 -- 172.21.15.23:0/1486336327 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c2c0a42b0 con 0x7f4c380f6610 2021-11-19T08:44:17.967 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.982+0000 7f4c3db92700 1 -- 172.21.15.23:0/1486336327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c380fdba0 con 0x7f4c380f6610 2021-11-19T08:44:17.967 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.983+0000 7f4c357fa700 1 -- 172.21.15.23:0/1486336327 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f4c2c04a800 con 0x7f4c380f6610 2021-11-19T08:44:17.968 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:17.968 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:09.251877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:17.970 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.984+0000 7f4c3db92700 1 -- 172.21.15.23:0/1486336327 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c2005c2b0 msgr2=0x7f4c2005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:17.970 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.984+0000 7f4c3db92700 1 --2- 172.21.15.23:0/1486336327 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c2005c2b0 0x7f4c2005e770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f4c380600d0 tx=0x7f4c3000b040).stop 2021-11-19T08:44:17.970 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.984+0000 7f4c3db92700 1 -- 172.21.15.23:0/1486336327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f6610 msgr2=0x7f4c3810bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:17.970 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.984+0000 7f4c3db92700 1 --2- 172.21.15.23:0/1486336327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f6610 0x7f4c3810bf40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f4c2c009f50 tx=0x7f4c2c009b30).stop 2021-11-19T08:44:17.971 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.985+0000 7f4c3db92700 1 -- 172.21.15.23:0/1486336327 shutdown_connections 2021-11-19T08:44:17.971 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.985+0000 7f4c3db92700 1 --2- 172.21.15.23:0/1486336327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c380f5370 0x7f4c38109a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:17.971 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.985+0000 7f4c3db92700 1 --2- 172.21.15.23:0/1486336327 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c2005c2b0 0x7f4c2005e770 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:17.971 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.985+0000 7f4c3db92700 1 --2- 172.21.15.23:0/1486336327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c380f6610 0x7f4c3810bf40 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:17.971 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.985+0000 7f4c3db92700 1 -- 172.21.15.23:0/1486336327 >> 172.21.15.23:0/1486336327 conn(0x7f4c380f0ab0 msgr2=0x7f4c380fa060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:17.972 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.985+0000 7f4c3db92700 1 -- 172.21.15.23:0/1486336327 shutdown_connections 2021-11-19T08:44:17.972 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:17.985+0000 7f4c3db92700 1 -- 172.21.15.23:0/1486336327 wait complete. 2021-11-19T08:44:17.972 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 78 2021-11-19T08:44:17.981 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:09.251877+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:17.982 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:resolve 2021-11-19T08:44:18.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:18 smithi023 conmon[33490]: cluster 2021-11-19T08:44:16.887375+0000 mgr.smithi023.ehlpcl (mgr.14164) 308 : cluster [DBG] pgmap v282: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:18.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:18 smithi023 conmon[33490]: audit 2021-11-19T08:44:17.984041+0000 mon.smithi023 (mon.0) 819 : audit [DBG] from='client.? 172.21.15.23:0/1486336327' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:18.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:18 smithi134 conmon[36363]: cluster 2021-11-19T08:44:16.887375+0000 mgr.smithi023.ehlpcl 2021-11-19T08:44:18.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:18 smithi134 conmon[36363]: (mgr.14164) 308 : cluster [DBG] pgmap v282: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:18.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:18 smithi134 conmon[36363]: audit 2021-11-19T08:44:17.984041+0000 mon.smithi023 (mon.0) 819 : audit [DBG] from='client.? 172.21.15.23:0/1486336327' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:18.983 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:19.112 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.126+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/3598596847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cb80f6820 msgr2=0x7f2cb80f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:19.113 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.126+0000 7f2cbeb5d700 1 --2- 172.21.15.23:0/3598596847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cb80f6820 0x7f2cb80f6c80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f2cac004660 tx=0x7f2cac009b30).stop 2021-11-19T08:44:19.113 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.126+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/3598596847 shutdown_connections 2021-11-19T08:44:19.113 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.126+0000 7f2cbeb5d700 1 --2- 172.21.15.23:0/3598596847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cb80f6820 0x7f2cb80f6c80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:19.114 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.126+0000 7f2cbeb5d700 1 --2- 172.21.15.23:0/3598596847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cb80f5580 0x7f2cb80f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:19.114 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.126+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/3598596847 >> 172.21.15.23:0/3598596847 conn(0x7f2cb80f0ce0 msgr2=0x7f2cb80f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:19.114 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.126+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/3598596847 shutdown_connections 2021-11-19T08:44:19.114 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.126+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/3598596847 wait complete. 2021-11-19T08:44:19.115 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.126+0000 7f2cbeb5d700 1 Processor -- start 2021-11-19T08:44:19.115 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.126+0000 7f2cbeb5d700 1 -- start start 2021-11-19T08:44:19.115 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.127+0000 7f2cbeb5d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cb80f5580 0x7f2cb8106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:19.115 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.127+0000 7f2cbeb5d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cb80f6820 0x7f2cb8107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:19.116 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.127+0000 7f2cbeb5d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb8107940 con 0x7f2cb80f6820 2021-11-19T08:44:19.116 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.127+0000 7f2cbeb5d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb8107a80 con 0x7f2cb80f5580 2021-11-19T08:44:19.116 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.127+0000 7f2cb7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cb80f6820 0x7f2cb8107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:19.116 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.127+0000 7f2cb7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cb80f6820 0x7f2cb8107390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48362/0 (socket says 172.21.15.23:48362) 2021-11-19T08:44:19.117 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.127+0000 7f2cb7fff700 1 -- 172.21.15.23:0/2580025340 learned_addr learned my addr 172.21.15.23:0/2580025340 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:19.117 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.127+0000 7f2cbc8f9700 1 --2- 172.21.15.23:0/2580025340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cb80f5580 0x7f2cb8106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:19.117 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.127+0000 7f2cb7fff700 1 -- 172.21.15.23:0/2580025340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cb80f5580 msgr2=0x7f2cb8106ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:19.117 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.127+0000 7f2cb7fff700 1 --2- 172.21.15.23:0/2580025340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cb80f5580 0x7f2cb8106ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:19.117 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.127+0000 7f2cb7fff700 1 -- 172.21.15.23:0/2580025340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cac005040 con 0x7f2cb80f6820 2021-11-19T08:44:19.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.127+0000 7f2cb7fff700 1 --2- 172.21.15.23:0/2580025340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cb80f6820 0x7f2cb8107390 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f2cac009f50 tx=0x7f2cac009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:19.118 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.128+0000 7f2cb57fa700 1 -- 172.21.15.23:0/2580025340 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cac018070 con 0x7f2cb80f6820 2021-11-19T08:44:19.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.128+0000 7f2cb57fa700 1 -- 172.21.15.23:0/2580025340 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2cac013410 con 0x7f2cb80f6820 2021-11-19T08:44:19.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.128+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/2580025340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb810c4c0 con 0x7f2cb80f6820 2021-11-19T08:44:19.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.128+0000 7f2cb57fa700 1 -- 172.21.15.23:0/2580025340 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cac01c3c0 con 0x7f2cb80f6820 2021-11-19T08:44:19.119 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.128+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/2580025340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb810ca80 con 0x7f2cb80f6820 2021-11-19T08:44:19.121 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.129+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/2580025340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb80fdd10 con 0x7f2cb80f6820 2021-11-19T08:44:19.122 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.130+0000 7f2cb57fa700 1 -- 172.21.15.23:0/2580025340 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2cac01d420 con 0x7f2cb80f6820 2021-11-19T08:44:19.123 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.131+0000 7f2cb57fa700 1 --2- 172.21.15.23:0/2580025340 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2ca005c2a0 0x7f2ca005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:19.123 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.131+0000 7f2cb57fa700 1 -- 172.21.15.23:0/2580025340 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2cac077fe0 con 0x7f2cb80f6820 2021-11-19T08:44:19.123 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.131+0000 7f2cbc8f9700 1 --2- 172.21.15.23:0/2580025340 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2ca005c2a0 0x7f2ca005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:19.123 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.131+0000 7f2cbc8f9700 1 --2- 172.21.15.23:0/2580025340 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2ca005c2a0 0x7f2ca005e760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f2cb80f6680 tx=0x7f2cb0009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:19.124 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.135+0000 7f2cb57fa700 1 -- 172.21.15.23:0/2580025340 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2cac04a800 con 0x7f2cb80f6820 2021-11-19T08:44:19.322 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.338+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/2580025340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cb810d180 con 0x7f2cb80f6820 2021-11-19T08:44:19.323 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.338+0000 7f2cb57fa700 1 -- 172.21.15.23:0/2580025340 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f2cac04de20 con 0x7f2cb80f6820 2021-11-19T08:44:19.323 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:19.324 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:09.251877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:19.325 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.340+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/2580025340 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2ca005c2a0 msgr2=0x7f2ca005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:19.325 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.340+0000 7f2cbeb5d700 1 --2- 172.21.15.23:0/2580025340 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2ca005c2a0 0x7f2ca005e760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f2cb80f6680 tx=0x7f2cb0009040).stop 2021-11-19T08:44:19.325 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.340+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/2580025340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cb80f6820 msgr2=0x7f2cb8107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:19.325 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.340+0000 7f2cbeb5d700 1 --2- 172.21.15.23:0/2580025340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cb80f6820 0x7f2cb8107390 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f2cac009f50 tx=0x7f2cac009b30).stop 2021-11-19T08:44:19.326 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.340+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/2580025340 shutdown_connections 2021-11-19T08:44:19.326 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.340+0000 7f2cbeb5d700 1 --2- 172.21.15.23:0/2580025340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cb80f5580 0x7f2cb8106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:19.326 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.340+0000 7f2cbeb5d700 1 --2- 172.21.15.23:0/2580025340 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2ca005c2a0 0x7f2ca005e760 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:19.326 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.340+0000 7f2cbeb5d700 1 --2- 172.21.15.23:0/2580025340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2cb80f6820 0x7f2cb8107390 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:19.327 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.340+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/2580025340 >> 172.21.15.23:0/2580025340 conn(0x7f2cb80f0ce0 msgr2=0x7f2cb805faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:19.327 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.340+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/2580025340 shutdown_connections 2021-11-19T08:44:19.327 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:19.340+0000 7f2cbeb5d700 1 -- 172.21.15.23:0/2580025340 wait complete. 2021-11-19T08:44:19.327 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 78 2021-11-19T08:44:19.337 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:09.251877+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:19.338 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:resolve 2021-11-19T08:44:20.340 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:20.477 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.489+0000 7f26747a3700 1 -- 172.21.15.23:0/3013862340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c0f55a0 msgr2=0x7f266c0f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:20.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.489+0000 7f26747a3700 1 --2- 172.21.15.23:0/3013862340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c0f55a0 0x7f266c0f59c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f26640099e0 tx=0x7f2664004d10).stop 2021-11-19T08:44:20.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.489+0000 7f26747a3700 1 -- 172.21.15.23:0/3013862340 shutdown_connections 2021-11-19T08:44:20.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.489+0000 7f26747a3700 1 --2- 172.21.15.23:0/3013862340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f266c0f6840 0x7f266c0f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:20.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.489+0000 7f26747a3700 1 --2- 172.21.15.23:0/3013862340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c0f55a0 0x7f266c0f59c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:20.479 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.489+0000 7f26747a3700 1 -- 172.21.15.23:0/3013862340 >> 172.21.15.23:0/3013862340 conn(0x7f266c0f0ce0 msgr2=0x7f266c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:20.479 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.489+0000 7f26747a3700 1 -- 172.21.15.23:0/3013862340 shutdown_connections 2021-11-19T08:44:20.479 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.489+0000 7f26747a3700 1 -- 172.21.15.23:0/3013862340 wait complete. 2021-11-19T08:44:20.479 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.490+0000 7f26747a3700 1 Processor -- start 2021-11-19T08:44:20.480 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.490+0000 7f26747a3700 1 -- start start 2021-11-19T08:44:20.480 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.490+0000 7f26747a3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f266c0f55a0 0x7f266c05ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:20.480 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.490+0000 7f26747a3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c0f6840 0x7f266c10bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:20.480 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.490+0000 7f26747a3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c10c290 con 0x7f266c0f6840 2021-11-19T08:44:20.481 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.490+0000 7f26747a3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c10c3d0 con 0x7f266c0f55a0 2021-11-19T08:44:20.481 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.490+0000 7f2671d3e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c0f6840 0x7f266c10bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:20.481 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.490+0000 7f2671d3e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c0f6840 0x7f266c10bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48380/0 (socket says 172.21.15.23:48380) 2021-11-19T08:44:20.481 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.490+0000 7f2671d3e700 1 -- 172.21.15.23:0/4055957246 learned_addr learned my addr 172.21.15.23:0/4055957246 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:20.482 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.490+0000 7f267253f700 1 --2- 172.21.15.23:0/4055957246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f266c0f55a0 0x7f266c05ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:20.482 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.490+0000 7f2671d3e700 1 -- 172.21.15.23:0/4055957246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f266c0f55a0 msgr2=0x7f266c05ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:20.482 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.490+0000 7f2671d3e700 1 --2- 172.21.15.23:0/4055957246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f266c0f55a0 0x7f266c05ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:20.482 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.490+0000 7f2671d3e700 1 -- 172.21.15.23:0/4055957246 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2664009640 con 0x7f266c0f6840 2021-11-19T08:44:20.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.490+0000 7f2671d3e700 1 --2- 172.21.15.23:0/4055957246 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c0f6840 0x7f266c10bda0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f2660002700 tx=0x7f2660009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:20.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.491+0000 7f266affd700 1 -- 172.21.15.23:0/4055957246 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2660009320 con 0x7f266c0f6840 2021-11-19T08:44:20.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.491+0000 7f26747a3700 1 -- 172.21.15.23:0/4055957246 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266c10c630 con 0x7f266c0f6840 2021-11-19T08:44:20.484 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.491+0000 7f266affd700 1 -- 172.21.15.23:0/4055957246 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f266000f040 con 0x7f266c0f6840 2021-11-19T08:44:20.484 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.491+0000 7f266affd700 1 -- 172.21.15.23:0/4055957246 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26600147e0 con 0x7f266c0f6840 2021-11-19T08:44:20.484 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.491+0000 7f26747a3700 1 -- 172.21.15.23:0/4055957246 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c10cb70 con 0x7f266c0f6840 2021-11-19T08:44:20.484 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.492+0000 7f26747a3700 1 -- 172.21.15.23:0/4055957246 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266c027a00 con 0x7f266c0f6840 2021-11-19T08:44:20.485 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.492+0000 7f266affd700 1 -- 172.21.15.23:0/4055957246 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2660009480 con 0x7f266c0f6840 2021-11-19T08:44:20.485 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.492+0000 7f266affd700 1 --2- 172.21.15.23:0/4055957246 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f265405c2a0 0x7f265405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:20.485 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.493+0000 7f266affd700 1 -- 172.21.15.23:0/4055957246 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f2660076960 con 0x7f266c0f6840 2021-11-19T08:44:20.485 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.493+0000 7f267253f700 1 --2- 172.21.15.23:0/4055957246 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f265405c2a0 0x7f265405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:20.486 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.493+0000 7f267253f700 1 --2- 172.21.15.23:0/4055957246 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f265405c2a0 0x7f265405e760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f266400b620 tx=0x7f2664004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:20.486 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.497+0000 7f266affd700 1 -- 172.21.15.23:0/4055957246 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26600491c0 con 0x7f266c0f6840 2021-11-19T08:44:20.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:20 smithi023 conmon[33490]: cluster 2021-11-19T08:44:18.887941+0000 mgr.smithi023.ehlpcl (mgr.14164) 309 : cluster [DBG] pgmap v283: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:20.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:20 smithi023 conmon[33490]: audit 2021-11-19T08:44:19.339208+0000 mon.smithi023 (mon.0) 820 : audit [DBG] from='client.? 172.21.15.23:0/2580025340' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:20.674 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.689+0000 7f26747a3700 1 -- 172.21.15.23:0/4055957246 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f266c0fdd30 con 0x7f266c0f6840 2021-11-19T08:44:20.674 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.690+0000 7f266affd700 1 -- 172.21.15.23:0/4055957246 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f266004c7e0 con 0x7f266c0f6840 2021-11-19T08:44:20.675 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:20.675 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:09.251877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:20.676 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.691+0000 7f26747a3700 1 -- 172.21.15.23:0/4055957246 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f265405c2a0 msgr2=0x7f265405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:20.677 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.691+0000 7f26747a3700 1 --2- 172.21.15.23:0/4055957246 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f265405c2a0 0x7f265405e760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f266400b620 tx=0x7f2664004d10).stop 2021-11-19T08:44:20.677 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.691+0000 7f26747a3700 1 -- 172.21.15.23:0/4055957246 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c0f6840 msgr2=0x7f266c10bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:20.677 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.691+0000 7f26747a3700 1 --2- 172.21.15.23:0/4055957246 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c0f6840 0x7f266c10bda0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f2660002700 tx=0x7f2660009de0).stop 2021-11-19T08:44:20.678 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.691+0000 7f26747a3700 1 -- 172.21.15.23:0/4055957246 shutdown_connections 2021-11-19T08:44:20.678 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.691+0000 7f26747a3700 1 --2- 172.21.15.23:0/4055957246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f266c0f55a0 0x7f266c05ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:20.678 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.691+0000 7f26747a3700 1 --2- 172.21.15.23:0/4055957246 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f265405c2a0 0x7f265405e760 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:20.678 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.691+0000 7f26747a3700 1 --2- 172.21.15.23:0/4055957246 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c0f6840 0x7f266c10bda0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:20.679 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.691+0000 7f26747a3700 1 -- 172.21.15.23:0/4055957246 >> 172.21.15.23:0/4055957246 conn(0x7f266c0f0ce0 msgr2=0x7f266c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:20.679 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.691+0000 7f26747a3700 1 -- 172.21.15.23:0/4055957246 shutdown_connections 2021-11-19T08:44:20.679 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:20.691+0000 7f26747a3700 1 -- 172.21.15.23:0/4055957246 wait complete. 2021-11-19T08:44:20.679 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 78 2021-11-19T08:44:20.688 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:09.251877+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:20.688 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:resolve 2021-11-19T08:44:20.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:20 smithi134 conmon[36363]: cluster 2021-11-19T08:44:18.887941+0000 mgr.smithi023.ehlpcl (mgr.14164) 309 : cluster [DBG] pgmap v283: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:20.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:20 smithi134 conmon[36363]: audit 2021-11-19T08:44:19.339208+0000 mon.smithi023 (mon.0) 820 : audit [DBG] from='client.? 172.21.15.23:0/2580025340' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:21.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:21 smithi023 conmon[33490]: audit 2021-11-19T08:44:20.690804+0000 mon.smithi023 (mon 2021-11-19T08:44:21.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:21 smithi023 conmon[33490]: .0) 821 : audit [DBG] from='client.? 172.21.15.23:0/4055957246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:21.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:21 smithi023 conmon[33490]: audit 2021-11-19T08:44:20.921068+0000 mon.smithi023 (mon.0) 822 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:44:21.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:21 smithi023 conmon[33490]: audit 2021-11- 2021-11-19T08:44:21.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:21 smithi023 conmon[33490]: 19T08:44:20.922170+0000 mon.smithi023 ( 2021-11-19T08:44:21.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:21 smithi023 conmon[33490]: mon.0) 823 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:44:21.689 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:21.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:21 smithi134 conmon[36363]: audit 2021-11-19T08:44:20.690804+0000 mon.smithi023 (mon.0) 821 : audit [DBG] from='client.? 172.21.15.23:0/4055957246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:21.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:21 smithi134 conmon[36363]: audit 2021-11-19T08:44:20.921068+0000 mon.smithi023 (mon.0) 822 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:44:21.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:21 smithi134 conmon[36363]: audit 2021-11-19T08:44:20.922170+0000 mon.smithi023 (mon.0) 823 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:44:21.824 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.837+0000 7fca77374700 1 -- 172.21.15.23:0/3506092547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca700f5580 msgr2=0x7fca700f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.837+0000 7fca77374700 1 --2- 172.21.15.23:0/3506092547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca700f5580 0x7fca700f59a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fca68004660 tx=0x7fca68009b30).stop 2021-11-19T08:44:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.837+0000 7fca77374700 1 -- 172.21.15.23:0/3506092547 shutdown_connections 2021-11-19T08:44:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.837+0000 7fca77374700 1 --2- 172.21.15.23:0/3506092547 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca700f6820 0x7fca700f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:21.825 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.837+0000 7fca77374700 1 --2- 172.21.15.23:0/3506092547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca700f5580 0x7fca700f59a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:21.826 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.837+0000 7fca77374700 1 -- 172.21.15.23:0/3506092547 >> 172.21.15.23:0/3506092547 conn(0x7fca700f0ce0 msgr2=0x7fca700f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:21.826 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.837+0000 7fca77374700 1 -- 172.21.15.23:0/3506092547 shutdown_connections 2021-11-19T08:44:21.826 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.837+0000 7fca77374700 1 -- 172.21.15.23:0/3506092547 wait complete. 2021-11-19T08:44:21.826 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.838+0000 7fca77374700 1 Processor -- start 2021-11-19T08:44:21.826 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.838+0000 7fca77374700 1 -- start start 2021-11-19T08:44:21.827 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.838+0000 7fca77374700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca700f5580 0x7fca700f9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:21.827 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.838+0000 7fca77374700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca700f6820 0x7fca700f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:21.827 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.838+0000 7fca77374700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca700f9ab0 con 0x7fca700f5580 2021-11-19T08:44:21.827 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.838+0000 7fca77374700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca700f8050 con 0x7fca700f6820 2021-11-19T08:44:21.828 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.838+0000 7fca75110700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca700f5580 0x7fca700f9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:21.828 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.838+0000 7fca75110700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca700f5580 0x7fca700f9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48388/0 (socket says 172.21.15.23:48388) 2021-11-19T08:44:21.828 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.838+0000 7fca75110700 1 -- 172.21.15.23:0/3385931254 learned_addr learned my addr 172.21.15.23:0/3385931254 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:21.828 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.838+0000 7fca7490f700 1 --2- 172.21.15.23:0/3385931254 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca700f6820 0x7fca700f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:21.829 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.838+0000 7fca75110700 1 -- 172.21.15.23:0/3385931254 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca700f6820 msgr2=0x7fca700f7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:21.829 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.838+0000 7fca75110700 1 --2- 172.21.15.23:0/3385931254 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca700f6820 0x7fca700f7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:21.829 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.838+0000 7fca75110700 1 -- 172.21.15.23:0/3385931254 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca68005040 con 0x7fca700f5580 2021-11-19T08:44:21.829 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.838+0000 7fca7490f700 1 --2- 172.21.15.23:0/3385931254 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca700f6820 0x7fca700f7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:44:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.839+0000 7fca75110700 1 --2- 172.21.15.23:0/3385931254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca700f5580 0x7fca700f9500 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fca68004e30 tx=0x7fca68009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.839+0000 7fca6dffb700 1 -- 172.21.15.23:0/3385931254 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca68018070 con 0x7fca700f5580 2021-11-19T08:44:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.839+0000 7fca6dffb700 1 -- 172.21.15.23:0/3385931254 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca68013410 con 0x7fca700f5580 2021-11-19T08:44:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.839+0000 7fca77374700 1 -- 172.21.15.23:0/3385931254 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca700f8270 con 0x7fca700f5580 2021-11-19T08:44:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.839+0000 7fca6dffb700 1 -- 172.21.15.23:0/3385931254 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca6801c3c0 con 0x7fca700f5580 2021-11-19T08:44:21.831 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.839+0000 7fca77374700 1 -- 172.21.15.23:0/3385931254 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca700f87d0 con 0x7fca700f5580 2021-11-19T08:44:21.831 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.840+0000 7fca77374700 1 -- 172.21.15.23:0/3385931254 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca700fdd10 con 0x7fca700f5580 2021-11-19T08:44:21.831 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.840+0000 7fca6dffb700 1 -- 172.21.15.23:0/3385931254 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fca6801d420 con 0x7fca700f5580 2021-11-19T08:44:21.832 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.841+0000 7fca6dffb700 1 --2- 172.21.15.23:0/3385931254 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fca5805c2a0 0x7fca5805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:21.833 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.841+0000 7fca6dffb700 1 -- 172.21.15.23:0/3385931254 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fca68077ef0 con 0x7fca700f5580 2021-11-19T08:44:21.833 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.841+0000 7fca7490f700 1 --2- 172.21.15.23:0/3385931254 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fca5805c2a0 0x7fca5805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:21.833 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.841+0000 7fca7490f700 1 --2- 172.21.15.23:0/3385931254 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fca5805c2a0 0x7fca5805e760 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fca64009fd0 tx=0x7fca6400b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:21.833 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:21.845+0000 7fca6dffb700 1 -- 172.21.15.23:0/3385931254 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca6804a7d0 con 0x7fca700f5580 2021-11-19T08:44:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:22.040+0000 7fca77374700 1 -- 172.21.15.23:0/3385931254 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca7010f6f0 con 0x7fca700f5580 2021-11-19T08:44:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:22.041+0000 7fca6dffb700 1 -- 172.21.15.23:0/3385931254 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2951 (secure 0 0 0) 0x7fca6804ddf0 con 0x7fca700f5580 2021-11-19T08:44:22.026 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:22.026 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:09.251877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:22.027 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:22.042+0000 7fca77374700 1 -- 172.21.15.23:0/3385931254 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fca5805c2a0 msgr2=0x7fca5805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:22.042+0000 7fca77374700 1 --2- 172.21.15.23:0/3385931254 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fca5805c2a0 0x7fca5805e760 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fca64009fd0 tx=0x7fca6400b040).stop 2021-11-19T08:44:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:22.042+0000 7fca77374700 1 -- 172.21.15.23:0/3385931254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca700f5580 msgr2=0x7fca700f9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:22.042+0000 7fca77374700 1 --2- 172.21.15.23:0/3385931254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca700f5580 0x7fca700f9500 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fca68004e30 tx=0x7fca68009b30).stop 2021-11-19T08:44:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:22.042+0000 7fca77374700 1 -- 172.21.15.23:0/3385931254 shutdown_connections 2021-11-19T08:44:22.029 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:22.042+0000 7fca77374700 1 --2- 172.21.15.23:0/3385931254 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fca5805c2a0 0x7fca5805e760 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:22.029 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:22.042+0000 7fca77374700 1 --2- 172.21.15.23:0/3385931254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca700f5580 0x7fca700f9500 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:22.029 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:22.042+0000 7fca77374700 1 --2- 172.21.15.23:0/3385931254 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca700f6820 0x7fca700f7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:22.029 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:22.042+0000 7fca77374700 1 -- 172.21.15.23:0/3385931254 >> 172.21.15.23:0/3385931254 conn(0x7fca700f0ce0 msgr2=0x7fca70060260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:22.029 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:22.042+0000 7fca77374700 1 -- 172.21.15.23:0/3385931254 shutdown_connections 2021-11-19T08:44:22.030 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:22.042+0000 7fca77374700 1 -- 172.21.15.23:0/3385931254 wait complete. 2021-11-19T08:44:22.030 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 78 2021-11-19T08:44:22.039 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:09.251877+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:22.039 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:resolve 2021-11-19T08:44:22.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:22 smithi023 conmon[33490]: cluster 2021-11-19T08:44:20.888432 2021-11-19T08:44:22.882 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:22 smithi023 conmon[33490]: +0000 mgr.smithi023.ehlpcl (mgr.14164) 310 : cluster [DBG] pgmap v284: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:22.882 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:22 smithi023 conmon[33490]: audit 2021-11-19T08:44:22.041732+0000 mon.smithi023 (mon.0) 2021-11-19T08:44:22.882 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:22 smithi023 conmon[33490]: 824 : audit [DBG] from='client.? 172.21.15.23:0/3385931254' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:22.883 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:22 smithi134 conmon[36363]: cluster 2021-11-19T08:44:20.888432+0000 mgr.smithi023.ehlpcl (mgr.14164) 310 : cluster [DBG] pgmap v284: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:22.883 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:22 smithi134 conmon[36363]: audit 2021-11-19T08:44:22.041732+0000 mon.smithi023 (mon.0) 824 : audit [DBG] from='client.? 172.21.15.23:0/3385931254' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:23.040 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:23.175 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.188+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3502807049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79f40f6680 msgr2=0x7f79f40f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:23.175 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.188+0000 7f79fc6c4700 1 --2- 172.21.15.23:0/3502807049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79f40f6680 0x7f79f40f6aa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f79e4004660 tx=0x7f79e4009b30).stop 2021-11-19T08:44:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.188+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3502807049 shutdown_connections 2021-11-19T08:44:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.188+0000 7f79fc6c4700 1 --2- 172.21.15.23:0/3502807049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79f40f7920 0x7f79f40f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.188+0000 7f79fc6c4700 1 --2- 172.21.15.23:0/3502807049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79f40f6680 0x7f79f40f6aa0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:23.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.188+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3502807049 >> 172.21.15.23:0/3502807049 conn(0x7f79f40f1de0 msgr2=0x7f79f40f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:23.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.189+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3502807049 shutdown_connections 2021-11-19T08:44:23.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.189+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3502807049 wait complete. 2021-11-19T08:44:23.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.189+0000 7f79fc6c4700 1 Processor -- start 2021-11-19T08:44:23.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.189+0000 7f79fc6c4700 1 -- start start 2021-11-19T08:44:23.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.189+0000 7f79fc6c4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79f40f6680 0x7f79f4106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:23.178 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.189+0000 7f79fc6c4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79f40f7920 0x7f79f41073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:23.178 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.189+0000 7f79fc6c4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f4107950 con 0x7f79f40f7920 2021-11-19T08:44:23.178 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.189+0000 7f79fc6c4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f4107a90 con 0x7f79f40f6680 2021-11-19T08:44:23.178 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.189+0000 7f79f9c5f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79f40f7920 0x7f79f41073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:23.179 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.189+0000 7f79f9c5f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79f40f7920 0x7f79f41073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48400/0 (socket says 172.21.15.23:48400) 2021-11-19T08:44:23.179 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.189+0000 7f79f9c5f700 1 -- 172.21.15.23:0/3869413282 learned_addr learned my addr 172.21.15.23:0/3869413282 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:23.179 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.190+0000 7f79fa460700 1 --2- 172.21.15.23:0/3869413282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79f40f6680 0x7f79f4106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:23.179 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.190+0000 7f79f9c5f700 1 -- 172.21.15.23:0/3869413282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79f40f6680 msgr2=0x7f79f4106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.190+0000 7f79f9c5f700 1 --2- 172.21.15.23:0/3869413282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79f40f6680 0x7f79f4106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.190+0000 7f79f9c5f700 1 -- 172.21.15.23:0/3869413282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e4005040 con 0x7f79f40f7920 2021-11-19T08:44:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.190+0000 7f79f9c5f700 1 --2- 172.21.15.23:0/3869413282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79f40f7920 0x7f79f41073a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f79f000c530 tx=0x7f79f0007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:23.180 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.190+0000 7f79eaffd700 1 -- 172.21.15.23:0/3869413282 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79f000f650 con 0x7f79f40f7920 2021-11-19T08:44:23.181 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.190+0000 7f79eaffd700 1 -- 172.21.15.23:0/3869413282 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79f000fc60 con 0x7f79f40f7920 2021-11-19T08:44:23.181 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.190+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3869413282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f410c530 con 0x7f79f40f7920 2021-11-19T08:44:23.181 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.190+0000 7f79eaffd700 1 -- 172.21.15.23:0/3869413282 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79f0012d30 con 0x7f79f40f7920 2021-11-19T08:44:23.181 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.190+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3869413282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f410caf0 con 0x7f79f40f7920 2021-11-19T08:44:23.182 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.191+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3869413282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f4027a00 con 0x7f79f40f7920 2021-11-19T08:44:23.183 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.191+0000 7f79eaffd700 1 -- 172.21.15.23:0/3869413282 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f79f0010070 con 0x7f79f40f7920 2021-11-19T08:44:23.183 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.192+0000 7f79eaffd700 1 --2- 172.21.15.23:0/3869413282 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f79e005c2a0 0x7f79e005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:23.183 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.192+0000 7f79eaffd700 1 -- 172.21.15.23:0/3869413282 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f79f00760b0 con 0x7f79f40f7920 2021-11-19T08:44:23.184 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.192+0000 7f79fa460700 1 --2- 172.21.15.23:0/3869413282 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f79e005c2a0 0x7f79e005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:23.184 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.192+0000 7f79fa460700 1 --2- 172.21.15.23:0/3869413282 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f79e005c2a0 0x7f79e005e760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f79e4004e30 tx=0x7f79e4009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:23.184 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.196+0000 7f79eaffd700 1 -- 172.21.15.23:0/3869413282 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f79f00452b0 con 0x7f79f40f7920 2021-11-19T08:44:23.378 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.393+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3869413282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79f40fee10 con 0x7f79f40f7920 2021-11-19T08:44:23.378 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.394+0000 7f79eaffd700 1 -- 172.21.15.23:0/3869413282 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f79f0048910 con 0x7f79f40f7920 2021-11-19T08:44:23.379 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:23.379 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":73},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":77,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:09.251877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":14732},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14732":{"gid":14732,"name":"cephfs.smithi023.fuvzkr","rank":1,"incarnation":69,"state":"up:active","state_seq":8,"addr":"172.21.15.23:6827/4052910066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":4052910066},{"type":"v1","addr":"172.21.15.23:6827","nonce":4052910066}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:23.381 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.395+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3869413282 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f79e005c2a0 msgr2=0x7f79e005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:23.381 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.395+0000 7f79fc6c4700 1 --2- 172.21.15.23:0/3869413282 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f79e005c2a0 0x7f79e005e760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f79e4004e30 tx=0x7f79e4009b30).stop 2021-11-19T08:44:23.381 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.395+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3869413282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79f40f7920 msgr2=0x7f79f41073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:23.381 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.395+0000 7f79fc6c4700 1 --2- 172.21.15.23:0/3869413282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79f40f7920 0x7f79f41073a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f79f000c530 tx=0x7f79f0007e20).stop 2021-11-19T08:44:23.382 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.396+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3869413282 shutdown_connections 2021-11-19T08:44:23.382 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.396+0000 7f79fc6c4700 1 --2- 172.21.15.23:0/3869413282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79f40f6680 0x7f79f4106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:23.383 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.396+0000 7f79fc6c4700 1 --2- 172.21.15.23:0/3869413282 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f79e005c2a0 0x7f79e005e760 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:23.383 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.396+0000 7f79fc6c4700 1 --2- 172.21.15.23:0/3869413282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79f40f7920 0x7f79f41073a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:23.383 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.396+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3869413282 >> 172.21.15.23:0/3869413282 conn(0x7f79f40f1de0 msgr2=0x7f79f40fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:23.383 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.396+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3869413282 shutdown_connections 2021-11-19T08:44:23.384 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:23.396+0000 7f79fc6c4700 1 -- 172.21.15.23:0/3869413282 wait complete. 2021-11-19T08:44:23.384 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 78 2021-11-19T08:44:23.392 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:09.251877+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 54, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 14732}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14732': {'gid': 14732, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.23:6827/4052910066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 4052910066}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 4052910066}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:23.393 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:resolve 2021-11-19T08:44:24.393 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.541+0000 7f61f0425700 1 -- 172.21.15.23:0/1572434712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e80f6820 msgr2=0x7f61e80f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.541+0000 7f61f0425700 1 --2- 172.21.15.23:0/1572434712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e80f6820 0x7f61e80f6c80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f61d8004660 tx=0x7f61d8009b30).stop 2021-11-19T08:44:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.541+0000 7f61f0425700 1 -- 172.21.15.23:0/1572434712 shutdown_connections 2021-11-19T08:44:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.541+0000 7f61f0425700 1 --2- 172.21.15.23:0/1572434712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e80f6820 0x7f61e80f6c80 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.541+0000 7f61f0425700 1 --2- 172.21.15.23:0/1572434712 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61e80f5580 0x7f61e80f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.541+0000 7f61f0425700 1 -- 172.21.15.23:0/1572434712 >> 172.21.15.23:0/1572434712 conn(0x7f61e80f0ce0 msgr2=0x7f61e80f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.542+0000 7f61f0425700 1 -- 172.21.15.23:0/1572434712 shutdown_connections 2021-11-19T08:44:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.542+0000 7f61f0425700 1 -- 172.21.15.23:0/1572434712 wait complete. 2021-11-19T08:44:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.542+0000 7f61f0425700 1 Processor -- start 2021-11-19T08:44:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.542+0000 7f61f0425700 1 -- start start 2021-11-19T08:44:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.542+0000 7f61f0425700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e80f5580 0x7f61e8060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:24.531 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.542+0000 7f61f0425700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61e80f6820 0x7f61e810bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:24.531 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.542+0000 7f61f0425700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e810c100 con 0x7f61e80f5580 2021-11-19T08:44:24.531 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.542+0000 7f61f0425700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e810c240 con 0x7f61e80f6820 2021-11-19T08:44:24.531 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.542+0000 7f61ee1c1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e80f5580 0x7f61e8060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.542+0000 7f61ee1c1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e80f5580 0x7f61e8060510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48410/0 (socket says 172.21.15.23:48410) 2021-11-19T08:44:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.542+0000 7f61ee1c1700 1 -- 172.21.15.23:0/887743265 learned_addr learned my addr 172.21.15.23:0/887743265 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.542+0000 7f61ed9c0700 1 --2- 172.21.15.23:0/887743265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61e80f6820 0x7f61e810bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.543+0000 7f61ee1c1700 1 -- 172.21.15.23:0/887743265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61e80f6820 msgr2=0x7f61e810bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.543+0000 7f61ee1c1700 1 --2- 172.21.15.23:0/887743265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61e80f6820 0x7f61e810bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.543+0000 7f61ee1c1700 1 -- 172.21.15.23:0/887743265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d8005040 con 0x7f61e80f5580 2021-11-19T08:44:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.543+0000 7f61ee1c1700 1 --2- 172.21.15.23:0/887743265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e80f5580 0x7f61e8060510 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f61e400e7f0 tx=0x7f61e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.543+0000 7f61deffd700 1 -- 172.21.15.23:0/887743265 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61e40092e0 con 0x7f61e80f5580 2021-11-19T08:44:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.543+0000 7f61deffd700 1 -- 172.21.15.23:0/887743265 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61e400f040 con 0x7f61e80f5580 2021-11-19T08:44:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.543+0000 7f61f0425700 1 -- 172.21.15.23:0/887743265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e810c4a0 con 0x7f61e80f5580 2021-11-19T08:44:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.543+0000 7f61deffd700 1 -- 172.21.15.23:0/887743265 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61e4014750 con 0x7f61e80f5580 2021-11-19T08:44:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.543+0000 7f61f0425700 1 -- 172.21.15.23:0/887743265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e810ca60 con 0x7f61e80f5580 2021-11-19T08:44:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.544+0000 7f61f0425700 1 -- 172.21.15.23:0/887743265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e8027a00 con 0x7f61e80f5580 2021-11-19T08:44:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.544+0000 7f61deffd700 1 -- 172.21.15.23:0/887743265 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f61e401b030 con 0x7f61e80f5580 2021-11-19T08:44:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.545+0000 7f61deffd700 1 --2- 172.21.15.23:0/887743265 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61d405c2a0 0x7f61d405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.545+0000 7f61deffd700 1 -- 172.21.15.23:0/887743265 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f61e40763b0 con 0x7f61e80f5580 2021-11-19T08:44:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.545+0000 7f61ed9c0700 1 --2- 172.21.15.23:0/887743265 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61d405c2a0 0x7f61d405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.545+0000 7f61ed9c0700 1 --2- 172.21.15.23:0/887743265 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61d405c2a0 0x7f61d405e760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f61d8005010 tx=0x7f61d8016040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.549+0000 7f61deffd700 1 -- 172.21.15.23:0/887743265 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61e4048bc0 con 0x7f61e80f5580 2021-11-19T08:44:24.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: cluster 2021-11-19T08:44:22.888994+0000 mgr.smithi023.ehlpcl (mgr.14164) 311 : 2021-11-19T08:44:24.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: cluster [DBG] pgmap v285: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:44:24.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: audit 2021-11-19T08:44:23.394932+0000 mon.smithi023 (mon.0) 825 : audit [DBG] from='client.? 172.21.15.23:0/3869413282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:24.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: 2021-11-19T08:44:24.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: cluster 2021-11-19T08:44:24.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: 2021-11-19T08:44 2021-11-19T08:44:24.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: :23.476322+0000 mon.smithi023 (mon 2021-11-19T08:44:24.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: .0) 826 : cluster [INF] daemon mds.cephfs.smithi023.fuvzkr restarted 2021-11-19T08:44:24.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: 2021-11-19T08:44:24.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: cluster 2021-11-19T08:44 2021-11-19T08:44:24.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: :23.485344+0000 mon.smithi023 (mon.0) 2021-11-19T08:44:24.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: 827 : cluster [DBG] osdmap e55: 6 total, 6 up, 6 in 2021-11-19T08:44:24.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: cluster 2021-11-19T08:44:24.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: 2021-11 2021-11-19T08:44:24.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: -19T08:44:24.226688+0000 2021-11-19T08:44:24.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: mon.smithi023 (mon.0) 828 : cluster 2021-11-19T08:44:24.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: [INF] Standby daemon mds.cephfs.smithi134.opxllb assigned to filesystem cephfs as rank 1 2021-11-19T08:44:24.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: cluster 2021-11 2021-11-19T08:44:24.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: -19T08:44:24.231825+0000 mon.smithi023 (mon 2021-11-19T08:44:24.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: .0) 829 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3968768531,v1:172.21.15.23:6827/3968768531] up:boot 2021-11-19T08:44:24.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: cluster 2021 2021-11-19T08:44:24.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: -11-19T08:44:24.231899 2021-11-19T08:44:24.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 830 : cluster 2021-11-19T08:44:24.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi134.opxllb=up:replay} 2 up:standby 2021-11-19T08:44:24.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: audit 2021-11-19T08: 2021-11-19T08:44:24.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: 44:24.232357+0000 mon.smithi023 (mon.0) 2021-11-19T08:44:24.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:24 smithi023 conmon[33490]: 831 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.fuvzkr"}]: dispatch 2021-11-19T08:44:24.732 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.747+0000 7f61f0425700 1 -- 172.21.15.23:0/887743265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61e80fdd10 con 0x7f61e80f5580 2021-11-19T08:44:24.733 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.748+0000 7f61deffd700 1 -- 172.21.15.23:0/887743265 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f61e4019090 con 0x7f61e80f5580 2021-11-19T08:44:24.733 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:24.733 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":79},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":79,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:24.226796+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:replay","state_seq":3,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:24.735 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.749+0000 7f61f0425700 1 -- 172.21.15.23:0/887743265 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61d405c2a0 msgr2=0x7f61d405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:24.735 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.749+0000 7f61f0425700 1 --2- 172.21.15.23:0/887743265 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61d405c2a0 0x7f61d405e760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f61d8005010 tx=0x7f61d8016040).stop 2021-11-19T08:44:24.735 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.749+0000 7f61f0425700 1 -- 172.21.15.23:0/887743265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e80f5580 msgr2=0x7f61e8060510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:24.736 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.749+0000 7f61f0425700 1 --2- 172.21.15.23:0/887743265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e80f5580 0x7f61e8060510 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f61e400e7f0 tx=0x7f61e400b040).stop 2021-11-19T08:44:24.736 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.750+0000 7f61f0425700 1 -- 172.21.15.23:0/887743265 shutdown_connections 2021-11-19T08:44:24.737 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.750+0000 7f61f0425700 1 --2- 172.21.15.23:0/887743265 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f61d405c2a0 0x7f61d405e760 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:24.737 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.750+0000 7f61f0425700 1 --2- 172.21.15.23:0/887743265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e80f5580 0x7f61e8060510 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:24.737 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.750+0000 7f61f0425700 1 --2- 172.21.15.23:0/887743265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f61e80f6820 0x7f61e810bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:24.738 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.750+0000 7f61f0425700 1 -- 172.21.15.23:0/887743265 >> 172.21.15.23:0/887743265 conn(0x7f61e80f0ce0 msgr2=0x7f61e80fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:24.738 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.750+0000 7f61f0425700 1 -- 172.21.15.23:0/887743265 shutdown_connections 2021-11-19T08:44:24.738 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:24.750+0000 7f61f0425700 1 -- 172.21.15.23:0/887743265 wait complete. 2021-11-19T08:44:24.738 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 79 2021-11-19T08:44:24.746 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:24.226796+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:24.746 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:resolve 2021-11-19T08:44:24.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: cluster 2021-11-19T08:44:22.888994+0000 mgr.smithi023.ehlpcl (mgr.14164) 311 : cluster [DBG] 2021-11-19T08:44:24.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: pgmap v285: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:44:24.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: audit 2021-11-19T08:44:23.394932+0000 mon.smithi023 (mon.0) 825 : audit [DBG] from='client.? 172.21.15.23:0/3869413282' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:24.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: cluster 2021-11-19T08:44:24.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: 2021-11-19T08:44:23.476322+0000 mon.smithi023 (mon.0) 826 : cluster [INF] daemon mds.cephfs.smithi023.fuvzkr restarted 2021-11-19T08:44:24.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: 2021-11-19T08:44:24.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: cluster 2021-11-19T 2021-11-19T08:44:24.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: 08:44:23.485344+0000 mon.smithi023 2021-11-19T08:44:24.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: (mon.0) 827 : cluster [DBG] osdmap e55: 6 total, 6 up, 6 in 2021-11-19T08:44:24.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: cluster 2021-11- 2021-11-19T08:44:24.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: 19T08:44:24.226688+0000 mon.smithi023 (mon 2021-11-19T08:44:24.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: .0) 828 : cluster [INF] Standby daemon mds.cephfs.smithi134.opxllb assigned to filesystem cephfs as rank 1 2021-11-19T08:44:24.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: cluster 2021-11-19T08:44:24.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: 2021-11-19T08:44:24.231825 2021-11-19T08:44:24.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 829 : cluster [DBG] 2021-11-19T08:44:24.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: mds.? [v2:172.21.15.23:6826/3968768531,v1:172.21.15.23:6827/3968768531] up:boot 2021-11-19T08:44:24.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: cluster 2021-11- 2021-11-19T08:44:24.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: 19T08:44:24.231899 2021-11-19T08:44:24.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 830 : cluster [DBG] 2021-11-19T08:44:24.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi134.opxllb=up:replay} 2 up:standby 2021-11-19T08:44:24.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: audit 2021-11-19 2021-11-19T08:44:24.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: T08:44:24.232357+0000 mon.smithi023 ( 2021-11-19T08:44:24.794 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:24 smithi134 conmon[36363]: mon.0) 831 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.fuvzkr"}]: dispatch 2021-11-19T08:44:25.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:25 smithi023 conmon[33490]: audit 2021-11-19T08:44:24.749066+0000 mon.smithi023 (mon.0) 832 : audit [DBG] from='client.? 172.21.15.23:0/887743265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:25.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:25 smithi023 conmon[33490]: cluster 2021-11-19 2021-11-19T08:44:25.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:25 smithi023 conmon[33490]: T08:44:25.239602+0000 mon.smithi023 (mon.0) 833 : cluster [DBG] mds.? [v2:172.21.15.134:6826/3725515320,v1:172.21.15.134:6827/3725515320] up:resolve 2021-11-19T08:44:25.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:25 smithi023 conmon[33490]: cluster 2021-11-19T08:44:25.239652+0000 mon.smithi023 (mon.0) 834 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi134.opxllb=up:resolve} 2 up:standby 2021-11-19T08:44:25.748 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:25.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:25 smithi134 conmon[36363]: audit 2021-11-19T08:44:24.749066+0000 mon.smithi023 (mon.0) 832 : audit 2021-11-19T08:44:25.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:25 smithi134 conmon[36363]: [DBG] from='client.? 172.21.15.23:0/887743265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:25.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:25 smithi134 conmon[36363]: cluster 2021-11-19T08:44:25.239602+0000 mon.smithi023 (mon.0) 833 : cluster [DBG] mds.? [v2:172.21.15.134:6826/3725515320,v1:172.21.15.134:6827/3725515320] up:resolve 2021-11-19T08:44:25.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:25 smithi134 conmon[36363]: cluster 2021-11-19T08:44:25.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:25 smithi134 conmon[36363]: 2021-11-19T08:44:25 2021-11-19T08:44:25.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:25 smithi134 conmon[36363]: .239652+0000 mon.smithi023 (mon.0) 834 2021-11-19T08:44:25.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:25 smithi134 conmon[36363]: : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:resolve,1=cephfs.smithi134.opxllb=up:resolve} 2 up:standby 2021-11-19T08:44:25.884 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.896+0000 7fb1d372a700 1 -- 172.21.15.23:0/911901760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1cc0f5580 msgr2=0x7fb1cc0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:25.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.896+0000 7fb1d372a700 1 --2- 172.21.15.23:0/911901760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1cc0f5580 0x7fb1cc0f59a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fb1c8004660 tx=0x7fb1c8009b30).stop 2021-11-19T08:44:25.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.896+0000 7fb1d372a700 1 -- 172.21.15.23:0/911901760 shutdown_connections 2021-11-19T08:44:25.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.896+0000 7fb1d372a700 1 --2- 172.21.15.23:0/911901760 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1cc0f6820 0x7fb1cc0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:25.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.896+0000 7fb1d372a700 1 --2- 172.21.15.23:0/911901760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1cc0f5580 0x7fb1cc0f59a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:25.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.896+0000 7fb1d372a700 1 -- 172.21.15.23:0/911901760 >> 172.21.15.23:0/911901760 conn(0x7fb1cc0f0ce0 msgr2=0x7fb1cc0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:25.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.896+0000 7fb1d372a700 1 -- 172.21.15.23:0/911901760 shutdown_connections 2021-11-19T08:44:25.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.896+0000 7fb1d372a700 1 -- 172.21.15.23:0/911901760 wait complete. 2021-11-19T08:44:25.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.897+0000 7fb1d372a700 1 Processor -- start 2021-11-19T08:44:25.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.897+0000 7fb1d372a700 1 -- start start 2021-11-19T08:44:25.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.897+0000 7fb1d372a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1cc0f5580 0x7fb1cc0f9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:25.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.897+0000 7fb1d372a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1cc0f6820 0x7fb1cc0f8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:25.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.897+0000 7fb1d372a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc0fa2d0 con 0x7fb1cc0f6820 2021-11-19T08:44:25.888 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.897+0000 7fb1d372a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc0f8870 con 0x7fb1cc0f5580 2021-11-19T08:44:25.888 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.897+0000 7fb1d0cc5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1cc0f6820 0x7fb1cc0f8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:25.888 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.897+0000 7fb1d0cc5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1cc0f6820 0x7fb1cc0f8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48424/0 (socket says 172.21.15.23:48424) 2021-11-19T08:44:25.888 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.897+0000 7fb1d0cc5700 1 -- 172.21.15.23:0/3524849455 learned_addr learned my addr 172.21.15.23:0/3524849455 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:25.888 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.897+0000 7fb1d14c6700 1 --2- 172.21.15.23:0/3524849455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1cc0f5580 0x7fb1cc0f9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:25.889 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.898+0000 7fb1d0cc5700 1 -- 172.21.15.23:0/3524849455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1cc0f5580 msgr2=0x7fb1cc0f9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:25.889 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.898+0000 7fb1d0cc5700 1 --2- 172.21.15.23:0/3524849455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1cc0f5580 0x7fb1cc0f9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:25.889 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.898+0000 7fb1d0cc5700 1 -- 172.21.15.23:0/3524849455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c8005040 con 0x7fb1cc0f6820 2021-11-19T08:44:25.889 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.898+0000 7fb1d0cc5700 1 --2- 172.21.15.23:0/3524849455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1cc0f6820 0x7fb1cc0f8380 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fb1c00026d0 tx=0x7fb1c0009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:25.890 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.898+0000 7fb1bdffb700 1 -- 172.21.15.23:0/3524849455 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1c00079c0 con 0x7fb1cc0f6820 2021-11-19T08:44:25.890 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.898+0000 7fb1bdffb700 1 -- 172.21.15.23:0/3524849455 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1c000f040 con 0x7fb1cc0f6820 2021-11-19T08:44:25.890 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.898+0000 7fb1d372a700 1 -- 172.21.15.23:0/3524849455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1cc0f8af0 con 0x7fb1cc0f6820 2021-11-19T08:44:25.891 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.898+0000 7fb1bdffb700 1 -- 172.21.15.23:0/3524849455 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1c00148b0 con 0x7fb1cc0f6820 2021-11-19T08:44:25.891 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.898+0000 7fb1d372a700 1 -- 172.21.15.23:0/3524849455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1cc0f90b0 con 0x7fb1cc0f6820 2021-11-19T08:44:25.891 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.899+0000 7fb1d372a700 1 -- 172.21.15.23:0/3524849455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1cc0fdd10 con 0x7fb1cc0f6820 2021-11-19T08:44:25.891 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.899+0000 7fb1bdffb700 1 -- 172.21.15.23:0/3524849455 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb1c0007b20 con 0x7fb1cc0f6820 2021-11-19T08:44:25.892 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.900+0000 7fb1bdffb700 1 --2- 172.21.15.23:0/3524849455 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb1b805c2a0 0x7fb1b805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:25.892 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.900+0000 7fb1bdffb700 1 -- 172.21.15.23:0/3524849455 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb1c00762f0 con 0x7fb1cc0f6820 2021-11-19T08:44:25.892 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.900+0000 7fb1d14c6700 1 --2- 172.21.15.23:0/3524849455 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb1b805c2a0 0x7fb1b805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:25.892 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.900+0000 7fb1d14c6700 1 --2- 172.21.15.23:0/3524849455 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb1b805c2a0 0x7fb1b805e760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fb1c8009f50 tx=0x7fb1c8009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:25.893 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:25.904+0000 7fb1bdffb700 1 -- 172.21.15.23:0/3524849455 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1c0048c30 con 0x7fb1cc0f6820 2021-11-19T08:44:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:26.099+0000 7fb1d372a700 1 -- 172.21.15.23:0/3524849455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1cc10f510 con 0x7fb1cc0f6820 2021-11-19T08:44:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:26.100+0000 7fb1bdffb700 1 -- 172.21.15.23:0/3524849455 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+2954 (secure 0 0 0) 0x7fb1c004c250 con 0x7fb1cc0f6820 2021-11-19T08:44:26.085 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:26.085 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":79},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":80,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:25.234436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:resolve","state_seq":11,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:26.086 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:26.101+0000 7fb1d372a700 1 -- 172.21.15.23:0/3524849455 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb1b805c2a0 msgr2=0x7fb1b805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:26.087 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:26.101+0000 7fb1d372a700 1 --2- 172.21.15.23:0/3524849455 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb1b805c2a0 0x7fb1b805e760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fb1c8009f50 tx=0x7fb1c8009b30).stop 2021-11-19T08:44:26.088 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:26.101+0000 7fb1d372a700 1 -- 172.21.15.23:0/3524849455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1cc0f6820 msgr2=0x7fb1cc0f8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:26.088 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:26.101+0000 7fb1d372a700 1 --2- 172.21.15.23:0/3524849455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1cc0f6820 0x7fb1cc0f8380 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fb1c00026d0 tx=0x7fb1c0009de0).stop 2021-11-19T08:44:26.089 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:26.102+0000 7fb1d372a700 1 -- 172.21.15.23:0/3524849455 shutdown_connections 2021-11-19T08:44:26.089 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:26.102+0000 7fb1d372a700 1 --2- 172.21.15.23:0/3524849455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1cc0f5580 0x7fb1cc0f9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:26.089 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:26.102+0000 7fb1d372a700 1 --2- 172.21.15.23:0/3524849455 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb1b805c2a0 0x7fb1b805e760 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:26.089 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:26.102+0000 7fb1d372a700 1 --2- 172.21.15.23:0/3524849455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1cc0f6820 0x7fb1cc0f8380 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:26.090 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:26.102+0000 7fb1d372a700 1 -- 172.21.15.23:0/3524849455 >> 172.21.15.23:0/3524849455 conn(0x7fb1cc0f0ce0 msgr2=0x7fb1cc05fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:26.090 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:26.102+0000 7fb1d372a700 1 -- 172.21.15.23:0/3524849455 shutdown_connections 2021-11-19T08:44:26.090 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:26.102+0000 7fb1d372a700 1 -- 172.21.15.23:0/3524849455 wait complete. 2021-11-19T08:44:26.090 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 80 2021-11-19T08:44:26.098 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:25.234436+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:resolve', 'state_seq': 7, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:resolve', 'state_seq': 11, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:26.099 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:resolve 2021-11-19T08:44:26.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:26 smithi023 conmon[33490]: cluster 2021-11-19T08:44:24.889469+0000 mgr.smithi023.ehlpcl (mgr.14164) 312 : cluster [DBG] pgmap v287: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-19T08:44:26.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:26 smithi023 conmon[33490]: audit 2021-11-19T08:44:26.100730+0000 mon.smithi023 (mon.0) 835 : audit [DBG] from='client.? 172.21.15.23:0/3524849455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:26.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:26 smithi023 conmon[33490]: cluster 2021-11-19T08:44:26.241317+0000 mon.smithi023 (mon.0) 836 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3188238627,v1:172.21.15.23:6829/3188238627] up:reconnect 2021-11-19T08:44:26.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:26 smithi023 conmon[33490]: cluster 2021-11-19T08:44:26.241378+0000 mon.smithi023 (mon.0) 837 : cluster [DBG] mds.? [v2:172.21.15.134:6826/3725515320,v1:172.21.15.134:6827/3725515320] up:reconnect 2021-11-19T08:44:26.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:26 smithi023 conmon[33490]: cluster 2021-11-19T08:44:26.241463+0000 mon.smithi023 (mon.0) 838 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:reconnect,1=cephfs.smithi134.opxllb=up:reconnect} 2 up:standby 2021-11-19T08:44:26.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:26 smithi023 conmon[33490]: cluster 2021-11-19T08:44:26.265552+0000 mds.cephfs.smithi134.opxllb (mds.1) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249994 2021-11-19T08:44:26.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: cluster 2021-11-19T08:44:24.889469+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:44:26.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: ) 312 : cluster [DBG] pgmap v287: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-19T08:44:26.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: audit 2021-11-19T08:44:26.100730+0000 mon.smithi023 (mon.0 2021-11-19T08:44:26.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: ) 835 : audit [DBG] from='client.? 172.21.15.23:0/3524849455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:26.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: cluster 2021-11- 2021-11-19T08:44:26.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: 19T08:44:26.241317+0000 2021-11-19T08:44:26.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: mon.smithi023 (mon.0) 836 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3188238627,v1:172.21.15.23:6829/3188238627] up:reconnect 2021-11-19T08:44:26.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: 2021-11-19T08:44:26.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: cluster 2021-11-19T 2021-11-19T08:44:26.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: 08:44:26.241378+0000 mon.smithi023 ( 2021-11-19T08:44:26.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: mon.0) 837 : cluster [DBG] mds.? [v2:172.21.15.134:6826/3725515320,v1:172.21.15.134:6827/3725515320] up:reconnect 2021-11-19T08:44:26.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: cluster 2021-11-19 2021-11-19T08:44:26.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: T08:44:26.241463+0000 mon.smithi023 (mon.0 2021-11-19T08:44:26.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: ) 838 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:reconnect,1=cephfs.smithi134.opxllb=up:reconnect} 2 up:standby 2021-11-19T08:44:26.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: cluster 2021 2021-11-19T08:44:26.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: -11-19T08:44:26. 2021-11-19T08:44:26.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: 265552+0000 mds.cephfs.smithi134.opxllb (mds.1) 1 : cluster 2021-11-19T08:44:26.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:26 smithi134 conmon[36363]: [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249994 2021-11-19T08:44:27.100 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.250+0000 7f927f188700 1 -- 172.21.15.23:0/57937932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92780f5910 msgr2=0x7f92780f5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.250+0000 7f927f188700 1 --2- 172.21.15.23:0/57937932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92780f5910 0x7f92780f5d90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f9264004660 tx=0x7f9264009b30).stop 2021-11-19T08:44:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.250+0000 7f927f188700 1 -- 172.21.15.23:0/57937932 shutdown_connections 2021-11-19T08:44:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.250+0000 7f927f188700 1 --2- 172.21.15.23:0/57937932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92780f5910 0x7f92780f5d90 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.250+0000 7f927f188700 1 --2- 172.21.15.23:0/57937932 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92780f6bc0 0x7f92780f6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.250+0000 7f927f188700 1 -- 172.21.15.23:0/57937932 >> 172.21.15.23:0/57937932 conn(0x7f92780f04a0 msgr2=0x7f92780f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.250+0000 7f927f188700 1 -- 172.21.15.23:0/57937932 shutdown_connections 2021-11-19T08:44:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.251+0000 7f927f188700 1 -- 172.21.15.23:0/57937932 wait complete. 2021-11-19T08:44:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.251+0000 7f927f188700 1 Processor -- start 2021-11-19T08:44:27.240 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.251+0000 7f927f188700 1 -- start start 2021-11-19T08:44:27.241 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.251+0000 7f927f188700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92780f5910 0x7f92780f52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:27.241 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.251+0000 7f927f188700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92780f6bc0 0x7f927810bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:27.241 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.251+0000 7f927f188700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927810c1c0 con 0x7f92780f6bc0 2021-11-19T08:44:27.242 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.251+0000 7f927f188700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927810c300 con 0x7f92780f5910 2021-11-19T08:44:27.242 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.252+0000 7f9277fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92780f6bc0 0x7f927810bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:27.242 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.252+0000 7f9277fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92780f6bc0 0x7f927810bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48438/0 (socket says 172.21.15.23:48438) 2021-11-19T08:44:27.242 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.252+0000 7f9277fff700 1 -- 172.21.15.23:0/44170746 learned_addr learned my addr 172.21.15.23:0/44170746 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.252+0000 7f927cf24700 1 --2- 172.21.15.23:0/44170746 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92780f5910 0x7f92780f52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.252+0000 7f9277fff700 1 -- 172.21.15.23:0/44170746 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92780f5910 msgr2=0x7f92780f52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.252+0000 7f9277fff700 1 --2- 172.21.15.23:0/44170746 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92780f5910 0x7f92780f52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:27.243 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.252+0000 7f9277fff700 1 -- 172.21.15.23:0/44170746 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9264005040 con 0x7f92780f6bc0 2021-11-19T08:44:27.244 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.252+0000 7f9277fff700 1 --2- 172.21.15.23:0/44170746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92780f6bc0 0x7f927810bcd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f9264009f50 tx=0x7f9264009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:27.244 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.252+0000 7f92757fa700 1 -- 172.21.15.23:0/44170746 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9264018070 con 0x7f92780f6bc0 2021-11-19T08:44:27.244 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.252+0000 7f92757fa700 1 -- 172.21.15.23:0/44170746 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9264013410 con 0x7f92780f6bc0 2021-11-19T08:44:27.244 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.252+0000 7f92757fa700 1 -- 172.21.15.23:0/44170746 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f926401c3c0 con 0x7f92780f6bc0 2021-11-19T08:44:27.244 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.252+0000 7f927f188700 1 -- 172.21.15.23:0/44170746 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927810c500 con 0x7f92780f6bc0 2021-11-19T08:44:27.245 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.252+0000 7f927f188700 1 -- 172.21.15.23:0/44170746 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927810ca30 con 0x7f92780f6bc0 2021-11-19T08:44:27.245 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.253+0000 7f927f188700 1 -- 172.21.15.23:0/44170746 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9278027a00 con 0x7f92780f6bc0 2021-11-19T08:44:27.246 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.254+0000 7f92757fa700 1 -- 172.21.15.23:0/44170746 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f926401d420 con 0x7f92780f6bc0 2021-11-19T08:44:27.246 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.254+0000 7f92757fa700 1 --2- 172.21.15.23:0/44170746 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f926805c2a0 0x7f926805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:27.247 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.255+0000 7f92757fa700 1 -- 172.21.15.23:0/44170746 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f92640786e0 con 0x7f92780f6bc0 2021-11-19T08:44:27.247 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.255+0000 7f927cf24700 1 --2- 172.21.15.23:0/44170746 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f926805c2a0 0x7f926805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:27.247 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.255+0000 7f927cf24700 1 --2- 172.21.15.23:0/44170746 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f926805c2a0 0x7f926805e760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f927805f7c0 tx=0x7f926c009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:27.247 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.258+0000 7f92757fa700 1 -- 172.21.15.23:0/44170746 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f926404afe0 con 0x7f92780f6bc0 2021-11-19T08:44:27.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:27 smithi023 conmon[33490]: cluster 2021-11-19T08:44:26.264637+0000 mds.cephfs.smithi023.wazanl (mds.0 2021-11-19T08:44:27.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:27 smithi023 conmon[33490]: ) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249997 2021-11-19T08:44:27.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:27 smithi023 conmon[33490]: cluster 2021-11-19T08:44:27.245466+0000 mon.smithi023 (mon.0) 839 : cluster [DBG] 2021-11-19T08:44:27.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:27 smithi023 conmon[33490]: mds.? [v2:172.21.15.23:6828/3188238627,v1:172.21.15.23:6829/3188238627] up:rejoin 2021-11-19T08:44:27.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:27 smithi023 conmon[33490]: cluster 2021-11-19T08:44:27.245532+0000 mon.smithi023 ( 2021-11-19T08:44:27.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:27 smithi023 conmon[33490]: mon.0) 840 : cluster [DBG] mds.? [v2:172.21.15.134:6826/3725515320,v1:172.21.15.134:6827/3725515320] up:rejoin 2021-11-19T08:44:27.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:27 smithi023 conmon[33490]: cluster 2021-11-19T08:44:27.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:27 smithi023 conmon[33490]: 2021-11-19T08:44:27.245607+0000 mon.smithi023 (mon.0) 841 : cluster [DBG] 2021-11-19T08:44:27.393 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:27 smithi023 conmon[33490]: fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:rejoin,1=cephfs.smithi134.opxllb=up:rejoin} 2 up:standby 2021-11-19T08:44:27.464 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.479+0000 7f927f188700 1 -- 172.21.15.23:0/44170746 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92780fdd90 con 0x7f92780f6bc0 2021-11-19T08:44:27.464 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.479+0000 7f92757fa700 1 -- 172.21.15.23:0/44170746 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2953 (secure 0 0 0) 0x7f926404e600 con 0x7f92780f6bc0 2021-11-19T08:44:27.464 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:27.465 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":79},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:27.239456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:27.466 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.481+0000 7f927f188700 1 -- 172.21.15.23:0/44170746 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f926805c2a0 msgr2=0x7f926805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:27.466 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.481+0000 7f927f188700 1 --2- 172.21.15.23:0/44170746 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f926805c2a0 0x7f926805e760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f927805f7c0 tx=0x7f926c009040).stop 2021-11-19T08:44:27.467 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.481+0000 7f927f188700 1 -- 172.21.15.23:0/44170746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92780f6bc0 msgr2=0x7f927810bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:27.467 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.481+0000 7f927f188700 1 --2- 172.21.15.23:0/44170746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92780f6bc0 0x7f927810bcd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f9264009f50 tx=0x7f9264009b30).stop 2021-11-19T08:44:27.467 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.481+0000 7f927f188700 1 -- 172.21.15.23:0/44170746 shutdown_connections 2021-11-19T08:44:27.467 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.481+0000 7f927f188700 1 --2- 172.21.15.23:0/44170746 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92780f5910 0x7f92780f52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:27.468 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.481+0000 7f927f188700 1 --2- 172.21.15.23:0/44170746 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f926805c2a0 0x7f926805e760 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:27.468 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.481+0000 7f927f188700 1 --2- 172.21.15.23:0/44170746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92780f6bc0 0x7f927810bcd0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:27.468 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.482+0000 7f927f188700 1 -- 172.21.15.23:0/44170746 >> 172.21.15.23:0/44170746 conn(0x7f92780f04a0 msgr2=0x7f92780fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:27.468 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.482+0000 7f927f188700 1 -- 172.21.15.23:0/44170746 shutdown_connections 2021-11-19T08:44:27.469 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:27.482+0000 7f927f188700 1 -- 172.21.15.23:0/44170746 wait complete. 2021-11-19T08:44:27.469 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 82 2021-11-19T08:44:27.478 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:27.239456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:27.478 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:rejoin 2021-11-19T08:44:27.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:27 smithi134 conmon[36363]: cluster 2021-11-19T08:44:26.264637+0000 mds.cephfs.smithi023.wazanl (mds.0) 1 2021-11-19T08:44:27.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:27 smithi134 conmon[36363]: : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249997 2021-11-19T08:44:27.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:27 smithi134 conmon[36363]: cluster 2021-11-19T08:44:27.245466+0000 mon.smithi023 (mon.0) 839 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3188238627,v1:172.21.15.23:6829/3188238627] up:rejoin 2021-11-19T08:44:27.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:27 smithi134 conmon[36363]: cluster 2021-11-19T08:44:27.245532+0000 mon.smithi023 (mon.0) 840 : cluster 2021-11-19T08:44:27.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:27 smithi134 conmon[36363]: [DBG] mds.? [v2:172.21.15.134:6826/3725515320,v1:172.21.15.134:6827/3725515320] up:rejoin 2021-11-19T08:44:27.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:27 smithi134 conmon[36363]: cluster 2021-11-19T08: 2021-11-19T08:44:27.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:27 smithi134 conmon[36363]: 44:27.245607+0000 mon.smithi023 (mon.0) 2021-11-19T08:44:27.793 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:27 smithi134 conmon[36363]: 841 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:rejoin,1=cephfs.smithi134.opxllb=up:rejoin} 2 up:standby 2021-11-19T08:44:28.480 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:28.512 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:28 smithi023 conmon[33490]: cluster 2021-11-19T08:44:26.890078+0000 mgr.smithi023.ehlpcl (mgr.14164) 313 : cluster [DBG] pgmap v288: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:28.512 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:28 smithi023 conmon[33490]: audit 2021-11-19T08:44:27.480557+0000 mon.smithi023 (mon.0) 842 : audit [DBG] from='client.? 172.21.15.23:0/44170746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:28.621 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.634+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3318585508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2f00f4d40 msgr2=0x7fa2f00f5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:28.621 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.634+0000 7fa2f77d0700 1 --2- 172.21.15.23:0/3318585508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2f00f4d40 0x7fa2f00f5160 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fa2e800aaf0 tx=0x7fa2e8004d10).stop 2021-11-19T08:44:28.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.634+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3318585508 shutdown_connections 2021-11-19T08:44:28.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.634+0000 7fa2f77d0700 1 --2- 172.21.15.23:0/3318585508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f00f5fe0 0x7fa2f00f6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:28.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.634+0000 7fa2f77d0700 1 --2- 172.21.15.23:0/3318585508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2f00f4d40 0x7fa2f00f5160 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:28.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.634+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3318585508 >> 172.21.15.23:0/3318585508 conn(0x7fa2f00f04a0 msgr2=0x7fa2f00f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:28.624 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.634+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3318585508 shutdown_connections 2021-11-19T08:44:28.624 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.634+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3318585508 wait complete. 2021-11-19T08:44:28.625 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.634+0000 7fa2f77d0700 1 Processor -- start 2021-11-19T08:44:28.625 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f77d0700 1 -- start start 2021-11-19T08:44:28.626 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f77d0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2f00f4d40 0x7fa2f0104c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:28.626 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f77d0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f00f5fe0 0x7fa2f0105140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:28.626 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f77d0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f0105740 con 0x7fa2f00f4d40 2021-11-19T08:44:28.627 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f77d0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f01058b0 con 0x7fa2f00f5fe0 2021-11-19T08:44:28.627 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f556c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2f00f4d40 0x7fa2f0104c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:28.627 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f556c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2f00f4d40 0x7fa2f0104c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48450/0 (socket says 172.21.15.23:48450) 2021-11-19T08:44:28.627 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f556c700 1 -- 172.21.15.23:0/3730806284 learned_addr learned my addr 172.21.15.23:0/3730806284 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:28.628 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f4d6b700 1 --2- 172.21.15.23:0/3730806284 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f00f5fe0 0x7fa2f0105140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:28.628 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f556c700 1 -- 172.21.15.23:0/3730806284 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f00f5fe0 msgr2=0x7fa2f0105140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:28.628 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f556c700 1 --2- 172.21.15.23:0/3730806284 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f00f5fe0 0x7fa2f0105140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:28.628 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f556c700 1 -- 172.21.15.23:0/3730806284 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e800a750 con 0x7fa2f00f4d40 2021-11-19T08:44:28.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f4d6b700 1 --2- 172.21.15.23:0/3730806284 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f00f5fe0 0x7fa2f0105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:44:28.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f556c700 1 --2- 172.21.15.23:0/3730806284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2f00f4d40 0x7fa2f0104c50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fa2e8004750 tx=0x7fa2e8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:28.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2e5ffb700 1 -- 172.21.15.23:0/3730806284 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2e800f030 con 0x7fa2f00f4d40 2021-11-19T08:44:28.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2e5ffb700 1 -- 172.21.15.23:0/3730806284 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2e801e8d0 con 0x7fa2f00f4d40 2021-11-19T08:44:28.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3730806284 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f010a2c0 con 0x7fa2f00f4d40 2021-11-19T08:44:28.630 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.635+0000 7fa2e5ffb700 1 -- 172.21.15.23:0/3730806284 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2e8019070 con 0x7fa2f00f4d40 2021-11-19T08:44:28.630 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.636+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3730806284 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f010a7f0 con 0x7fa2f00f4d40 2021-11-19T08:44:28.631 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.637+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3730806284 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f0046330 con 0x7fa2f00f4d40 2021-11-19T08:44:28.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.637+0000 7fa2e5ffb700 1 -- 172.21.15.23:0/3730806284 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa2e8025060 con 0x7fa2f00f4d40 2021-11-19T08:44:28.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.637+0000 7fa2e5ffb700 1 --2- 172.21.15.23:0/3730806284 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa2dc05c2b0 0x7fa2dc05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:28.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.638+0000 7fa2e5ffb700 1 -- 172.21.15.23:0/3730806284 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fa2e8078ae0 con 0x7fa2f00f4d40 2021-11-19T08:44:28.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.638+0000 7fa2f4d6b700 1 --2- 172.21.15.23:0/3730806284 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa2dc05c2b0 0x7fa2dc05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:28.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.638+0000 7fa2f4d6b700 1 --2- 172.21.15.23:0/3730806284 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa2dc05c2b0 0x7fa2dc05e770 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fa2e0009fd0 tx=0x7fa2e000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:28.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.642+0000 7fa2e5ffb700 1 -- 172.21.15.23:0/3730806284 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2e804b2f0 con 0x7fa2f00f4d40 2021-11-19T08:44:28.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:28 smithi134 conmon[36363]: cluster 2021-11-19T08:44:26.890078+0000 mgr.smithi023.ehlpcl (mgr.14164) 313 : cluster [DBG] 2021-11-19T08:44:28.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:28 smithi134 conmon[36363]: pgmap v288: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:28.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:28 smithi134 conmon[36363]: audit 2021-11-19T08:44:27.480557+0000 mon.smithi023 (mon.0) 842 : audit [DBG] from='client.? 172.21.15.23:0/44170746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:28.822 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.837+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3730806284 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2f0027a00 con 0x7fa2f00f4d40 2021-11-19T08:44:28.822 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.838+0000 7fa2e5ffb700 1 -- 172.21.15.23:0/3730806284 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2952 (secure 0 0 0) 0x7fa2e804e910 con 0x7fa2f00f4d40 2021-11-19T08:44:28.823 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:28.823 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:27.239456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.839+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3730806284 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa2dc05c2b0 msgr2=0x7fa2dc05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.839+0000 7fa2f77d0700 1 --2- 172.21.15.23:0/3730806284 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa2dc05c2b0 0x7fa2dc05e770 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fa2e0009fd0 tx=0x7fa2e000b040).stop 2021-11-19T08:44:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.839+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3730806284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2f00f4d40 msgr2=0x7fa2f0104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.839+0000 7fa2f77d0700 1 --2- 172.21.15.23:0/3730806284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2f00f4d40 0x7fa2f0104c50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fa2e8004750 tx=0x7fa2e8004d10).stop 2021-11-19T08:44:28.826 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.839+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3730806284 shutdown_connections 2021-11-19T08:44:28.826 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.839+0000 7fa2f77d0700 1 --2- 172.21.15.23:0/3730806284 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa2dc05c2b0 0x7fa2dc05e770 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:28.826 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.839+0000 7fa2f77d0700 1 --2- 172.21.15.23:0/3730806284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2f00f4d40 0x7fa2f0104c50 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:28.826 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.839+0000 7fa2f77d0700 1 --2- 172.21.15.23:0/3730806284 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2f00f5fe0 0x7fa2f0105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:28.827 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.839+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3730806284 >> 172.21.15.23:0/3730806284 conn(0x7fa2f00f04a0 msgr2=0x7fa2f00f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:28.827 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.840+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3730806284 shutdown_connections 2021-11-19T08:44:28.827 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:28.840+0000 7fa2f77d0700 1 -- 172.21.15.23:0/3730806284 wait complete. 2021-11-19T08:44:28.828 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 83 2021-11-19T08:44:28.836 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:27.239456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:28.837 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:rejoin 2021-11-19T08:44:29.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:29 smithi023 conmon[33490]: cluster 2021-11-19T08:44:28.350029+0000 mon.smithi023 ( 2021-11-19T08:44:29.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:29 smithi023 conmon[33490]: mon.0) 843 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3968768531,v1:172.21.15.23:6827/3968768531] up:standby 2021-11-19T08:44:29.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:29 smithi023 conmon[33490]: cluster 2021-11-19T08:44:28.350105+0000 mon.smithi023 (mon.0) 844 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:rejoin,1=cephfs.smithi134.opxllb=up:rejoin} 2 up:standby 2021-11-19T08:44:29.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:29 smithi023 conmon[33490]: audit 2021-11-19T08:44:28.838679+0000 mon.smithi023 (mon.0) 845 : audit [DBG] from='client.? 172.21.15.23:0/3730806284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:29.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:29 smithi134 conmon[36363]: cluster 2021-11-19T08:44:28.350029+0000 mon.smithi023 (mon.0) 843 : cluster [DBG] 2021-11-19T08:44:29.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:29 smithi134 conmon[36363]: mds.? [v2:172.21.15.23:6826/3968768531,v1:172.21.15.23:6827/3968768531] up:standby 2021-11-19T08:44:29.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:29 smithi134 conmon[36363]: cluster 2021-11-19T08:44:28.350105+0000 mon.smithi023 (mon.0) 844 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.wazanl=up:rejoin,1=cephfs.smithi134.opxllb=up:rejoin} 2 up:standby 2021-11-19T08:44:29.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:29 smithi134 conmon[36363]: audit 2021-11-19T 2021-11-19T08:44:29.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:29 smithi134 conmon[36363]: 08:44:28.838679+0000 mon.smithi023 (mon.0) 845 : audit [DBG] from='client.? 172.21.15.23:0/3730806284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:29.838 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:29.974 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.987+0000 7fe353ebd700 1 -- 172.21.15.23:0/1356349120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe34c0f6820 msgr2=0x7fe34c0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:29.975 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.987+0000 7fe353ebd700 1 --2- 172.21.15.23:0/1356349120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe34c0f6820 0x7fe34c0f6c80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fe340004660 tx=0x7fe340009b30).stop 2021-11-19T08:44:29.975 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.987+0000 7fe353ebd700 1 -- 172.21.15.23:0/1356349120 shutdown_connections 2021-11-19T08:44:29.975 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.987+0000 7fe353ebd700 1 --2- 172.21.15.23:0/1356349120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe34c0f6820 0x7fe34c0f6c80 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:29.975 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.987+0000 7fe353ebd700 1 --2- 172.21.15.23:0/1356349120 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe34c0f5580 0x7fe34c0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:29.976 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.987+0000 7fe353ebd700 1 -- 172.21.15.23:0/1356349120 >> 172.21.15.23:0/1356349120 conn(0x7fe34c0f0ce0 msgr2=0x7fe34c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:29.976 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.987+0000 7fe353ebd700 1 -- 172.21.15.23:0/1356349120 shutdown_connections 2021-11-19T08:44:29.976 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.987+0000 7fe353ebd700 1 -- 172.21.15.23:0/1356349120 wait complete. 2021-11-19T08:44:29.976 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.987+0000 7fe353ebd700 1 Processor -- start 2021-11-19T08:44:29.977 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.987+0000 7fe353ebd700 1 -- start start 2021-11-19T08:44:29.977 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe353ebd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe34c0f5580 0x7fe34c106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:29.977 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe353ebd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe34c0f6820 0x7fe34c1073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:29.977 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe353ebd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c107950 con 0x7fe34c0f6820 2021-11-19T08:44:29.977 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe353ebd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c107a90 con 0x7fe34c0f5580 2021-11-19T08:44:29.978 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe351458700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe34c0f6820 0x7fe34c1073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:29.978 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe351458700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe34c0f6820 0x7fe34c1073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48464/0 (socket says 172.21.15.23:48464) 2021-11-19T08:44:29.978 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe351458700 1 -- 172.21.15.23:0/580886961 learned_addr learned my addr 172.21.15.23:0/580886961 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:29.979 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe351c59700 1 --2- 172.21.15.23:0/580886961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe34c0f5580 0x7fe34c106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:29.979 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe351458700 1 -- 172.21.15.23:0/580886961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe34c0f5580 msgr2=0x7fe34c106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:29.979 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe351458700 1 --2- 172.21.15.23:0/580886961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe34c0f5580 0x7fe34c106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:29.979 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe351458700 1 -- 172.21.15.23:0/580886961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe340005040 con 0x7fe34c0f6820 2021-11-19T08:44:29.979 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe351c59700 1 --2- 172.21.15.23:0/580886961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe34c0f5580 0x7fe34c106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:44:29.980 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe351458700 1 --2- 172.21.15.23:0/580886961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe34c0f6820 0x7fe34c1073a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fe340009830 tx=0x7fe340009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:29.980 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe34a7fc700 1 -- 172.21.15.23:0/580886961 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe340018070 con 0x7fe34c0f6820 2021-11-19T08:44:29.980 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe353ebd700 1 -- 172.21.15.23:0/580886961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34c10c4d0 con 0x7fe34c0f6820 2021-11-19T08:44:29.980 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe34a7fc700 1 -- 172.21.15.23:0/580886961 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe340013410 con 0x7fe34c0f6820 2021-11-19T08:44:29.980 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe34a7fc700 1 -- 172.21.15.23:0/580886961 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe34001c620 con 0x7fe34c0f6820 2021-11-19T08:44:29.981 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.988+0000 7fe353ebd700 1 -- 172.21.15.23:0/580886961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34c10ca90 con 0x7fe34c0f6820 2021-11-19T08:44:29.981 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.989+0000 7fe353ebd700 1 -- 172.21.15.23:0/580886961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34c0fdd10 con 0x7fe34c0f6820 2021-11-19T08:44:29.981 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.990+0000 7fe34a7fc700 1 -- 172.21.15.23:0/580886961 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe34001c8e0 con 0x7fe34c0f6820 2021-11-19T08:44:29.981 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.990+0000 7fe34a7fc700 1 --2- 172.21.15.23:0/580886961 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe33405c2a0 0x7fe33405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:29.982 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.990+0000 7fe34a7fc700 1 -- 172.21.15.23:0/580886961 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fe340078250 con 0x7fe34c0f6820 2021-11-19T08:44:29.982 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.990+0000 7fe351c59700 1 --2- 172.21.15.23:0/580886961 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe33405c2a0 0x7fe33405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:29.982 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.991+0000 7fe351c59700 1 --2- 172.21.15.23:0/580886961 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe33405c2a0 0x7fe33405e760 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fe34c0f6680 tx=0x7fe344009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:29.983 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:29.994+0000 7fe34a7fc700 1 -- 172.21.15.23:0/580886961 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe34004aa00 con 0x7fe34c0f6820 2021-11-19T08:44:30.174 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:30.189+0000 7fe353ebd700 1 -- 172.21.15.23:0/580886961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe34c10d3a0 con 0x7fe34c0f6820 2021-11-19T08:44:30.175 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:30.190+0000 7fe34a7fc700 1 -- 172.21.15.23:0/580886961 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2952 (secure 0 0 0) 0x7fe34004e020 con 0x7fe34c0f6820 2021-11-19T08:44:30.175 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:30.176 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:27.239456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:30.191+0000 7fe353ebd700 1 -- 172.21.15.23:0/580886961 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe33405c2a0 msgr2=0x7fe33405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:30.191+0000 7fe353ebd700 1 --2- 172.21.15.23:0/580886961 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe33405c2a0 0x7fe33405e760 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fe34c0f6680 tx=0x7fe344009040).stop 2021-11-19T08:44:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:30.191+0000 7fe353ebd700 1 -- 172.21.15.23:0/580886961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe34c0f6820 msgr2=0x7fe34c1073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:30.191+0000 7fe353ebd700 1 --2- 172.21.15.23:0/580886961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe34c0f6820 0x7fe34c1073a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fe340009830 tx=0x7fe340009b30).stop 2021-11-19T08:44:30.180 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:30.191+0000 7fe353ebd700 1 -- 172.21.15.23:0/580886961 shutdown_connections 2021-11-19T08:44:30.180 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:30.191+0000 7fe353ebd700 1 --2- 172.21.15.23:0/580886961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe34c0f5580 0x7fe34c106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:30.180 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:30.191+0000 7fe353ebd700 1 --2- 172.21.15.23:0/580886961 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe33405c2a0 0x7fe33405e760 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:30.181 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:30.191+0000 7fe353ebd700 1 --2- 172.21.15.23:0/580886961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe34c0f6820 0x7fe34c1073a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:30.181 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:30.191+0000 7fe353ebd700 1 -- 172.21.15.23:0/580886961 >> 172.21.15.23:0/580886961 conn(0x7fe34c0f0ce0 msgr2=0x7fe34c060260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:30.181 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:30.191+0000 7fe353ebd700 1 -- 172.21.15.23:0/580886961 shutdown_connections 2021-11-19T08:44:30.181 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:30.191+0000 7fe353ebd700 1 -- 172.21.15.23:0/580886961 wait complete. 2021-11-19T08:44:30.182 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 83 2021-11-19T08:44:30.192 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:27.239456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:30.192 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:rejoin 2021-11-19T08:44:30.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:30 smithi023 conmon[33490]: cluster 2021-11-19T08:44:28.890497+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:44:30.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:30 smithi023 conmon[33490]: mgr.14164) 314 : cluster [DBG] pgmap v289: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:30.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:30 smithi023 conmon[33490]: audit 2021-11-19T08:44:30.190878+0000 mon.smithi023 ( 2021-11-19T08:44:30.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:30 smithi023 conmon[33490]: mon.0) 846 : audit [DBG] from='client.? 172.21.15.23:0/580886961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:30.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:30 smithi134 conmon[36363]: cluster 2021-11-19T08:44:28.890497+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:44:30.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:30 smithi134 conmon[36363]: ) 314 : cluster [DBG] pgmap v289: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:30.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:30 smithi134 conmon[36363]: audit 2021-11-19T08:44:30.190878+0000 mon.smithi023 (mon.0) 846 : audit [DBG] from='client.? 172.21.15.23:0/580886961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:31.194 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:31.329 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.342+0000 7f96ad699700 1 -- 172.21.15.23:0/545730725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96a80f55a0 msgr2=0x7f96a80f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:31.329 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.342+0000 7f96ad699700 1 --2- 172.21.15.23:0/545730725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96a80f55a0 0x7f96a80f59c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f96a00099e0 tx=0x7f96a0004d10).stop 2021-11-19T08:44:31.329 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.342+0000 7f96ad699700 1 -- 172.21.15.23:0/545730725 shutdown_connections 2021-11-19T08:44:31.330 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.342+0000 7f96ad699700 1 --2- 172.21.15.23:0/545730725 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96a80f6840 0x7f96a80f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:31.330 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.342+0000 7f96ad699700 1 --2- 172.21.15.23:0/545730725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96a80f55a0 0x7f96a80f59c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:31.330 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.342+0000 7f96ad699700 1 -- 172.21.15.23:0/545730725 >> 172.21.15.23:0/545730725 conn(0x7f96a80f0ce0 msgr2=0x7f96a80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:31.330 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.342+0000 7f96ad699700 1 -- 172.21.15.23:0/545730725 shutdown_connections 2021-11-19T08:44:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.342+0000 7f96ad699700 1 -- 172.21.15.23:0/545730725 wait complete. 2021-11-19T08:44:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.342+0000 7f96ad699700 1 Processor -- start 2021-11-19T08:44:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f96ad699700 1 -- start start 2021-11-19T08:44:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f96ad699700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96a80f55a0 0x7f96a805ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f96ad699700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96a80f6840 0x7f96a810e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f96ad699700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a8107ba0 con 0x7f96a80f6840 2021-11-19T08:44:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f96ad699700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a810e6b0 con 0x7f96a80f55a0 2021-11-19T08:44:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f96a67fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96a80f6840 0x7f96a810e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f96a67fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96a80f6840 0x7f96a810e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48474/0 (socket says 172.21.15.23:48474) 2021-11-19T08:44:31.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f96a67fc700 1 -- 172.21.15.23:0/1159101628 learned_addr learned my addr 172.21.15.23:0/1159101628 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:31.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f96a6ffd700 1 --2- 172.21.15.23:0/1159101628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96a80f55a0 0x7f96a805ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:31.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f96a67fc700 1 -- 172.21.15.23:0/1159101628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96a80f55a0 msgr2=0x7f96a805ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:31.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f96a67fc700 1 --2- 172.21.15.23:0/1159101628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96a80f55a0 0x7f96a805ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:31.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f96a67fc700 1 -- 172.21.15.23:0/1159101628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a0009640 con 0x7f96a80f6840 2021-11-19T08:44:31.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f96a67fc700 1 --2- 172.21.15.23:0/1159101628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96a80f6840 0x7f96a810e1c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f969000c530 tx=0x7f9690007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:31.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f969f7fe700 1 -- 172.21.15.23:0/1159101628 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f969000f650 con 0x7f96a80f6840 2021-11-19T08:44:31.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f96ad699700 1 -- 172.21.15.23:0/1159101628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a810e850 con 0x7f96a80f6840 2021-11-19T08:44:31.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f969f7fe700 1 -- 172.21.15.23:0/1159101628 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f969000fc60 con 0x7f96a80f6840 2021-11-19T08:44:31.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.343+0000 7f969f7fe700 1 -- 172.21.15.23:0/1159101628 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9690012c80 con 0x7f96a80f6840 2021-11-19T08:44:31.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.344+0000 7f96ad699700 1 -- 172.21.15.23:0/1159101628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a810ed60 con 0x7f96a80f6840 2021-11-19T08:44:31.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.345+0000 7f96ad699700 1 -- 172.21.15.23:0/1159101628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96a8046330 con 0x7f96a80f6840 2021-11-19T08:44:31.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.345+0000 7f969f7fe700 1 -- 172.21.15.23:0/1159101628 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9690010070 con 0x7f96a80f6840 2021-11-19T08:44:31.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.345+0000 7f969f7fe700 1 --2- 172.21.15.23:0/1159101628 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f969405c2a0 0x7f969405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:31.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.345+0000 7f969f7fe700 1 -- 172.21.15.23:0/1159101628 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f9690076170 con 0x7f96a80f6840 2021-11-19T08:44:31.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.345+0000 7f96a6ffd700 1 --2- 172.21.15.23:0/1159101628 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f969405c2a0 0x7f969405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:31.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.346+0000 7f96a6ffd700 1 --2- 172.21.15.23:0/1159101628 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f969405c2a0 0x7f969405e760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f96a0000f80 tx=0x7f96a0004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:31.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.349+0000 7f969f7fe700 1 -- 172.21.15.23:0/1159101628 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96900452b0 con 0x7f96a80f6840 2021-11-19T08:44:31.530 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.546+0000 7f96ad699700 1 -- 172.21.15.23:0/1159101628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96a8027a00 con 0x7f96a80f6840 2021-11-19T08:44:31.531 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.546+0000 7f969f7fe700 1 -- 172.21.15.23:0/1159101628 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f969004c2f0 con 0x7f96a80f6840 2021-11-19T08:44:31.532 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:31.532 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:27.239456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:31.533 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.548+0000 7f96ad699700 1 -- 172.21.15.23:0/1159101628 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f969405c2a0 msgr2=0x7f969405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:31.534 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.548+0000 7f96ad699700 1 --2- 172.21.15.23:0/1159101628 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f969405c2a0 0x7f969405e760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f96a0000f80 tx=0x7f96a0004d10).stop 2021-11-19T08:44:31.534 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.548+0000 7f96ad699700 1 -- 172.21.15.23:0/1159101628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96a80f6840 msgr2=0x7f96a810e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:31.534 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.548+0000 7f96ad699700 1 --2- 172.21.15.23:0/1159101628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96a80f6840 0x7f96a810e1c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f969000c530 tx=0x7f9690007e20).stop 2021-11-19T08:44:31.534 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.548+0000 7f96ad699700 1 -- 172.21.15.23:0/1159101628 shutdown_connections 2021-11-19T08:44:31.535 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.548+0000 7f96ad699700 1 --2- 172.21.15.23:0/1159101628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96a80f55a0 0x7f96a805ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:31.535 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.548+0000 7f96ad699700 1 --2- 172.21.15.23:0/1159101628 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f969405c2a0 0x7f969405e760 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:31.535 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.548+0000 7f96ad699700 1 --2- 172.21.15.23:0/1159101628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96a80f6840 0x7f96a810e1c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:31.535 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.548+0000 7f96ad699700 1 -- 172.21.15.23:0/1159101628 >> 172.21.15.23:0/1159101628 conn(0x7f96a80f0ce0 msgr2=0x7f96a80fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:31.535 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.548+0000 7f96ad699700 1 -- 172.21.15.23:0/1159101628 shutdown_connections 2021-11-19T08:44:31.536 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:31.548+0000 7f96ad699700 1 -- 172.21.15.23:0/1159101628 wait complete. 2021-11-19T08:44:31.536 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 83 2021-11-19T08:44:31.545 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:27.239456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:31.545 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:rejoin 2021-11-19T08:44:32.546 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:32.575 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:32 smithi023 conmon[33490]: cluster 2021-11-19T08:44:30.891226+0000 mgr.smithi023.ehlpcl (mgr.14164) 315 : cluster [DBG] pgmap v290: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:32.576 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:32 smithi023 conmon[33490]: audit 2021-11-19T08:44:31.547393+0000 mon.smithi023 (mon.0) 847 : audit [DBG] from='client.? 172.21.15.23:0/1159101628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:32.681 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.694+0000 7fb36ebf2700 1 -- 172.21.15.23:0/4143227830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb36806b940 msgr2=0x7fb368069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:32.681 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.694+0000 7fb36ebf2700 1 --2- 172.21.15.23:0/4143227830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb36806b940 0x7fb368069e40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fb364004660 tx=0x7fb364009b30).stop 2021-11-19T08:44:32.681 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.694+0000 7fb36ebf2700 1 -- 172.21.15.23:0/4143227830 shutdown_connections 2021-11-19T08:44:32.681 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.694+0000 7fb36ebf2700 1 --2- 172.21.15.23:0/4143227830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb36806a330 0x7fb36806a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:32.682 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.694+0000 7fb36ebf2700 1 --2- 172.21.15.23:0/4143227830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb36806b940 0x7fb368069e40 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:32.682 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.694+0000 7fb36ebf2700 1 -- 172.21.15.23:0/4143227830 >> 172.21.15.23:0/4143227830 conn(0x7fb3680f1d30 msgr2=0x7fb3680f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:32.682 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.694+0000 7fb36ebf2700 1 -- 172.21.15.23:0/4143227830 shutdown_connections 2021-11-19T08:44:32.682 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.694+0000 7fb36ebf2700 1 -- 172.21.15.23:0/4143227830 wait complete. 2021-11-19T08:44:32.683 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.694+0000 7fb36ebf2700 1 Processor -- start 2021-11-19T08:44:32.683 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.694+0000 7fb36ebf2700 1 -- start start 2021-11-19T08:44:32.683 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.694+0000 7fb36ebf2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb36806a330 0x7fb368108f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:32.683 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.694+0000 7fb36ebf2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb36806b940 0x7fb368109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:32.683 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb36ebf2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3681099e0 con 0x7fb36806b940 2021-11-19T08:44:32.684 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb36ebf2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb368109b20 con 0x7fb36806a330 2021-11-19T08:44:32.684 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb36d3ef700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb36806b940 0x7fb368109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:32.684 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb36d3ef700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb36806b940 0x7fb368109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48486/0 (socket says 172.21.15.23:48486) 2021-11-19T08:44:32.684 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb36d3ef700 1 -- 172.21.15.23:0/221018619 learned_addr learned my addr 172.21.15.23:0/221018619 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:32.685 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb36dbf0700 1 --2- 172.21.15.23:0/221018619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb36806a330 0x7fb368108f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:32.685 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb36d3ef700 1 -- 172.21.15.23:0/221018619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb36806a330 msgr2=0x7fb368108f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:32.685 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb36d3ef700 1 --2- 172.21.15.23:0/221018619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb36806a330 0x7fb368108f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:32.685 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb36d3ef700 1 -- 172.21.15.23:0/221018619 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb364005040 con 0x7fb36806b940 2021-11-19T08:44:32.685 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb36dbf0700 1 --2- 172.21.15.23:0/221018619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb36806a330 0x7fb368108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:44:32.686 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb36d3ef700 1 --2- 172.21.15.23:0/221018619 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb36806b940 0x7fb368109430 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fb3580026d0 tx=0x7fb358009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:32.686 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb35e7fc700 1 -- 172.21.15.23:0/221018619 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb358009320 con 0x7fb36806b940 2021-11-19T08:44:32.686 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb35e7fc700 1 -- 172.21.15.23:0/221018619 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb35800f040 con 0x7fb36806b940 2021-11-19T08:44:32.686 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb36ebf2700 1 -- 172.21.15.23:0/221018619 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36810a4c0 con 0x7fb36806b940 2021-11-19T08:44:32.687 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb35e7fc700 1 -- 172.21.15.23:0/221018619 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3580147e0 con 0x7fb36806b940 2021-11-19T08:44:32.688 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.695+0000 7fb36ebf2700 1 -- 172.21.15.23:0/221018619 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36810aa80 con 0x7fb36806b940 2021-11-19T08:44:32.688 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.696+0000 7fb36ebf2700 1 -- 172.21.15.23:0/221018619 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb368045ad0 con 0x7fb36806b940 2021-11-19T08:44:32.688 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.696+0000 7fb35e7fc700 1 -- 172.21.15.23:0/221018619 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb3580077b0 con 0x7fb36806b940 2021-11-19T08:44:32.689 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.697+0000 7fb35e7fc700 1 --2- 172.21.15.23:0/221018619 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb35405c300 0x7fb35405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:32.689 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.697+0000 7fb35e7fc700 1 -- 172.21.15.23:0/221018619 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7fb358076980 con 0x7fb36806b940 2021-11-19T08:44:32.690 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.697+0000 7fb36dbf0700 1 --2- 172.21.15.23:0/221018619 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb35405c300 0x7fb35405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:32.690 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.697+0000 7fb36dbf0700 1 --2- 172.21.15.23:0/221018619 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb35405c300 0x7fb35405e7c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fb364009d60 tx=0x7fb364009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:32.690 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.701+0000 7fb35e7fc700 1 -- 172.21.15.23:0/221018619 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb358049210 con 0x7fb36806b940 2021-11-19T08:44:32.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:32 smithi134 conmon[36363]: cluster 2021-11-19T08:44:30.891226+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:44:32.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:32 smithi134 conmon[36363]: ) 315 : cluster [DBG] pgmap v290: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:32.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:32 smithi134 conmon[36363]: audit 2021-11-19T08:44:31.547393+0000 mon.smithi023 (mon.0) 847 : audit [DBG] from='client.? 172.21.15.23:0/1159101628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:32.880 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.895+0000 7fb36ebf2700 1 -- 172.21.15.23:0/221018619 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb368027a00 con 0x7fb36806b940 2021-11-19T08:44:32.880 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.895+0000 7fb35e7fc700 1 -- 172.21.15.23:0/221018619 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2952 (secure 0 0 0) 0x7fb35804c830 con 0x7fb36806b940 2021-11-19T08:44:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:32.881 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:27.239456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:32.882 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.897+0000 7fb36ebf2700 1 -- 172.21.15.23:0/221018619 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb35405c300 msgr2=0x7fb35405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:32.883 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.897+0000 7fb36ebf2700 1 --2- 172.21.15.23:0/221018619 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb35405c300 0x7fb35405e7c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fb364009d60 tx=0x7fb364009b30).stop 2021-11-19T08:44:32.883 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.897+0000 7fb36ebf2700 1 -- 172.21.15.23:0/221018619 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb36806b940 msgr2=0x7fb368109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:32.883 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.897+0000 7fb36ebf2700 1 --2- 172.21.15.23:0/221018619 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb36806b940 0x7fb368109430 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fb3580026d0 tx=0x7fb358009de0).stop 2021-11-19T08:44:32.883 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.897+0000 7fb36ebf2700 1 -- 172.21.15.23:0/221018619 shutdown_connections 2021-11-19T08:44:32.884 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.897+0000 7fb36ebf2700 1 --2- 172.21.15.23:0/221018619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb36806a330 0x7fb368108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:32.884 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.897+0000 7fb36ebf2700 1 --2- 172.21.15.23:0/221018619 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fb35405c300 0x7fb35405e7c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:32.884 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.897+0000 7fb36ebf2700 1 --2- 172.21.15.23:0/221018619 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb36806b940 0x7fb368109430 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:32.884 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.897+0000 7fb36ebf2700 1 -- 172.21.15.23:0/221018619 >> 172.21.15.23:0/221018619 conn(0x7fb3680f1d30 msgr2=0x7fb3680fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:32.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.897+0000 7fb36ebf2700 1 -- 172.21.15.23:0/221018619 shutdown_connections 2021-11-19T08:44:32.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:32.897+0000 7fb36ebf2700 1 -- 172.21.15.23:0/221018619 wait complete. 2021-11-19T08:44:32.885 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 83 2021-11-19T08:44:32.893 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:27.239456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:32.894 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:rejoin 2021-11-19T08:44:33.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:33 smithi023 conmon[33490]: audit 2021-11-19T08:44:32.896369+0000 mon.smithi023 (mon.0) 848 : audit [DBG] from='client.? 172.21.15.23:0/221018619' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:33.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:33 smithi134 conmon[36363]: audit 2021-11-19T08:44:32.896369+0000 mon.smithi023 (mon.0) 848 : 2021-11-19T08:44:33.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:33 smithi134 conmon[36363]: audit [DBG] from='client.? 172.21.15.23:0/221018619' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:33.895 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:34.030 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.043+0000 7f1b5359e700 1 -- 172.21.15.23:0/1885479157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b540f6490 msgr2=0x7f1b540f68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:34.030 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.043+0000 7f1b5359e700 1 --2- 172.21.15.23:0/1885479157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b540f6490 0x7f1b540f68b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f1b44004660 tx=0x7f1b44009b30).stop 2021-11-19T08:44:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b5359e700 1 -- 172.21.15.23:0/1885479157 shutdown_connections 2021-11-19T08:44:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b5359e700 1 --2- 172.21.15.23:0/1885479157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1b540f7730 0x7f1b540f7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b5359e700 1 --2- 172.21.15.23:0/1885479157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b540f6490 0x7f1b540f68b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b5359e700 1 -- 172.21.15.23:0/1885479157 >> 172.21.15.23:0/1885479157 conn(0x7f1b540f1bd0 msgr2=0x7f1b540f4010 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b5359e700 1 -- 172.21.15.23:0/1885479157 shutdown_connections 2021-11-19T08:44:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b5359e700 1 -- 172.21.15.23:0/1885479157 wait complete. 2021-11-19T08:44:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b5359e700 1 Processor -- start 2021-11-19T08:44:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b5359e700 1 -- start start 2021-11-19T08:44:34.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b5359e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1b540f6490 0x7f1b5406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:34.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b5359e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b540f7730 0x7f1b54069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:34.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b5359e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5406a380 con 0x7f1b540f7730 2021-11-19T08:44:34.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b5359e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5406a4c0 con 0x7f1b540f6490 2021-11-19T08:44:34.033 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b51d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b540f7730 0x7f1b54069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:34.034 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b51d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b540f7730 0x7f1b54069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48496/0 (socket says 172.21.15.23:48496) 2021-11-19T08:44:34.034 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.044+0000 7f1b51d9b700 1 -- 172.21.15.23:0/2970770265 learned_addr learned my addr 172.21.15.23:0/2970770265 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:34.034 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.045+0000 7f1b5259c700 1 --2- 172.21.15.23:0/2970770265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1b540f6490 0x7f1b5406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:34.034 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.045+0000 7f1b51d9b700 1 -- 172.21.15.23:0/2970770265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1b540f6490 msgr2=0x7f1b5406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:34.035 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.045+0000 7f1b51d9b700 1 --2- 172.21.15.23:0/2970770265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1b540f6490 0x7f1b5406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:34.035 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.045+0000 7f1b51d9b700 1 -- 172.21.15.23:0/2970770265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b44005040 con 0x7f1b540f7730 2021-11-19T08:44:34.036 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.045+0000 7f1b51d9b700 1 --2- 172.21.15.23:0/2970770265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b540f7730 0x7f1b54069e20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f1b480026d0 tx=0x7f1b48009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:34.036 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.045+0000 7f1b42ffd700 1 -- 172.21.15.23:0/2970770265 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b480079c0 con 0x7f1b540f7730 2021-11-19T08:44:34.036 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.045+0000 7f1b42ffd700 1 -- 172.21.15.23:0/2970770265 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b4800f040 con 0x7f1b540f7730 2021-11-19T08:44:34.036 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.045+0000 7f1b5359e700 1 -- 172.21.15.23:0/2970770265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5406a7a0 con 0x7f1b540f7730 2021-11-19T08:44:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.045+0000 7f1b42ffd700 1 -- 172.21.15.23:0/2970770265 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b480148b0 con 0x7f1b540f7730 2021-11-19T08:44:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.045+0000 7f1b5359e700 1 -- 172.21.15.23:0/2970770265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5410eda0 con 0x7f1b540f7730 2021-11-19T08:44:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.046+0000 7f1b5359e700 1 -- 172.21.15.23:0/2970770265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b54045ad0 con 0x7f1b540f7730 2021-11-19T08:44:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.046+0000 7f1b42ffd700 1 -- 172.21.15.23:0/2970770265 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1b48007b20 con 0x7f1b540f7730 2021-11-19T08:44:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.047+0000 7f1b42ffd700 1 --2- 172.21.15.23:0/2970770265 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f1b3c05c2a0 0x7f1b3c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.047+0000 7f1b42ffd700 1 -- 172.21.15.23:0/2970770265 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f1b480762f0 con 0x7f1b540f7730 2021-11-19T08:44:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.047+0000 7f1b5259c700 1 --2- 172.21.15.23:0/2970770265 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f1b3c05c2a0 0x7f1b3c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.047+0000 7f1b5259c700 1 --2- 172.21.15.23:0/2970770265 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f1b3c05c2a0 0x7f1b3c05e760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f1b44005010 tx=0x7f1b44009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.051+0000 7f1b42ffd700 1 -- 172.21.15.23:0/2970770265 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b48048b00 con 0x7f1b540f7730 2021-11-19T08:44:34.233 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.248+0000 7f1b5359e700 1 -- 172.21.15.23:0/2970770265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b54027a00 con 0x7f1b540f7730 2021-11-19T08:44:34.233 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.248+0000 7f1b42ffd700 1 -- 172.21.15.23:0/2970770265 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f1b4804c120 con 0x7f1b540f7730 2021-11-19T08:44:34.233 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:34.234 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:27.239456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:34.235 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.250+0000 7f1b5359e700 1 -- 172.21.15.23:0/2970770265 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f1b3c05c2a0 msgr2=0x7f1b3c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:34.235 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.250+0000 7f1b5359e700 1 --2- 172.21.15.23:0/2970770265 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f1b3c05c2a0 0x7f1b3c05e760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f1b44005010 tx=0x7f1b44009b30).stop 2021-11-19T08:44:34.235 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.250+0000 7f1b5359e700 1 -- 172.21.15.23:0/2970770265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b540f7730 msgr2=0x7f1b54069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:34.235 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.250+0000 7f1b5359e700 1 --2- 172.21.15.23:0/2970770265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b540f7730 0x7f1b54069e20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f1b480026d0 tx=0x7f1b48009de0).stop 2021-11-19T08:44:34.236 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.250+0000 7f1b5359e700 1 -- 172.21.15.23:0/2970770265 shutdown_connections 2021-11-19T08:44:34.236 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.250+0000 7f1b5359e700 1 --2- 172.21.15.23:0/2970770265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1b540f6490 0x7f1b5406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:34.236 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.250+0000 7f1b5359e700 1 --2- 172.21.15.23:0/2970770265 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f1b3c05c2a0 0x7f1b3c05e760 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:34.236 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.250+0000 7f1b5359e700 1 --2- 172.21.15.23:0/2970770265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b540f7730 0x7f1b54069e20 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:34.237 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.250+0000 7f1b5359e700 1 -- 172.21.15.23:0/2970770265 >> 172.21.15.23:0/2970770265 conn(0x7f1b540f1bd0 msgr2=0x7f1b540fb180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:34.237 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.250+0000 7f1b5359e700 1 -- 172.21.15.23:0/2970770265 shutdown_connections 2021-11-19T08:44:34.237 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:34.250+0000 7f1b5359e700 1 -- 172.21.15.23:0/2970770265 wait complete. 2021-11-19T08:44:34.238 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 83 2021-11-19T08:44:34.247 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:27.239456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:34.247 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:rejoin 2021-11-19T08:44:34.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:34 smithi023 conmon[33490]: cluster 2021-11-19T08:44:32.891571+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:44:34.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:34 smithi023 conmon[33490]: .14164) 316 : cluster [DBG] pgmap v291: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:34.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:34 smithi023 conmon[33490]: audit 2021-11-19T08:44:34.249507 2021-11-19T08:44:34.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:34 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 849 : audit [DBG] from='client.? 172.21.15.23:0/2970770265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:34.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:34 smithi023 conmon[33490]: 2021-11-19T08:44:34.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:34 smithi134 conmon[36363]: cluster 2021-11-19T08:44:32.891571+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:44:34.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:34 smithi134 conmon[36363]: 14164) 316 : cluster [DBG] pgmap v291: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:34.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:34 smithi134 conmon[36363]: audit 2021-11-19T08:44:34.249507+0000 mon.smithi023 (mon.0) 849 : audit 2021-11-19T08:44:34.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:34 smithi134 conmon[36363]: [DBG] from='client.? 172.21.15.23:0/2970770265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:35.248 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:35.383 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.396+0000 7f2f5359e700 1 -- 172.21.15.23:0/3903971631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f540f5660 msgr2=0x7f2f540f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:35.384 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.396+0000 7f2f5359e700 1 --2- 172.21.15.23:0/3903971631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f540f5660 0x7f2f540f5ae0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f2f48004660 tx=0x7f2f48009b30).stop 2021-11-19T08:44:35.384 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.396+0000 7f2f5359e700 1 -- 172.21.15.23:0/3903971631 shutdown_connections 2021-11-19T08:44:35.384 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.396+0000 7f2f5359e700 1 --2- 172.21.15.23:0/3903971631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f540f5660 0x7f2f540f5ae0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:35.384 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.396+0000 7f2f5359e700 1 --2- 172.21.15.23:0/3903971631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f540f6960 0x7f2f540f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:35.385 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.396+0000 7f2f5359e700 1 -- 172.21.15.23:0/3903971631 >> 172.21.15.23:0/3903971631 conn(0x7f2f540f0230 msgr2=0x7f2f540f2690 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:35.385 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.397+0000 7f2f5359e700 1 -- 172.21.15.23:0/3903971631 shutdown_connections 2021-11-19T08:44:35.385 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.397+0000 7f2f5359e700 1 -- 172.21.15.23:0/3903971631 wait complete. 2021-11-19T08:44:35.385 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.397+0000 7f2f5359e700 1 Processor -- start 2021-11-19T08:44:35.386 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.397+0000 7f2f5359e700 1 -- start start 2021-11-19T08:44:35.386 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.397+0000 7f2f5359e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f540f5660 0x7f2f54106d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:35.386 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.397+0000 7f2f5359e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f540f6960 0x7f2f54107280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:35.386 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.397+0000 7f2f5359e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f54107880 con 0x7f2f540f5660 2021-11-19T08:44:35.386 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.397+0000 7f2f5359e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f541080c0 con 0x7f2f540f6960 2021-11-19T08:44:35.387 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.397+0000 7f2f5259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f540f5660 0x7f2f54106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:35.387 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.397+0000 7f2f5259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f540f5660 0x7f2f54106d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48504/0 (socket says 172.21.15.23:48504) 2021-11-19T08:44:35.387 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.397+0000 7f2f5259c700 1 -- 172.21.15.23:0/1654348015 learned_addr learned my addr 172.21.15.23:0/1654348015 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:35.387 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.397+0000 7f2f51d9b700 1 --2- 172.21.15.23:0/1654348015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f540f6960 0x7f2f54107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:35.388 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.398+0000 7f2f5259c700 1 -- 172.21.15.23:0/1654348015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f540f6960 msgr2=0x7f2f54107280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:35.388 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.398+0000 7f2f5259c700 1 --2- 172.21.15.23:0/1654348015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f540f6960 0x7f2f54107280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:35.388 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.398+0000 7f2f5259c700 1 -- 172.21.15.23:0/1654348015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f48005040 con 0x7f2f540f5660 2021-11-19T08:44:35.388 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.398+0000 7f2f51d9b700 1 --2- 172.21.15.23:0/1654348015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f540f6960 0x7f2f54107280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:44:35.389 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.398+0000 7f2f5259c700 1 --2- 172.21.15.23:0/1654348015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f540f5660 0x7f2f54106d90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f2f44002700 tx=0x7f2f44009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:35.389 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.398+0000 7f2f42ffd700 1 -- 172.21.15.23:0/1654348015 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f44009320 con 0x7f2f540f5660 2021-11-19T08:44:35.389 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.398+0000 7f2f42ffd700 1 -- 172.21.15.23:0/1654348015 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f4400f040 con 0x7f2f540f5660 2021-11-19T08:44:35.389 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.398+0000 7f2f5359e700 1 -- 172.21.15.23:0/1654348015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f541083a0 con 0x7f2f540f5660 2021-11-19T08:44:35.389 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.398+0000 7f2f42ffd700 1 -- 172.21.15.23:0/1654348015 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f44014900 con 0x7f2f540f5660 2021-11-19T08:44:35.390 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.398+0000 7f2f5359e700 1 -- 172.21.15.23:0/1654348015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f541088e0 con 0x7f2f540f5660 2021-11-19T08:44:35.390 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.399+0000 7f2f5359e700 1 -- 172.21.15.23:0/1654348015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f540f3f00 con 0x7f2f540f5660 2021-11-19T08:44:35.390 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.399+0000 7f2f42ffd700 1 -- 172.21.15.23:0/1654348015 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2f44009480 con 0x7f2f540f5660 2021-11-19T08:44:35.391 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.400+0000 7f2f42ffd700 1 --2- 172.21.15.23:0/1654348015 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2f3c05c2b0 0x7f2f3c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:35.391 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.400+0000 7f2f42ffd700 1 -- 172.21.15.23:0/1654348015 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2f440766c0 con 0x7f2f540f5660 2021-11-19T08:44:35.392 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.400+0000 7f2f51d9b700 1 --2- 172.21.15.23:0/1654348015 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2f3c05c2b0 0x7f2f3c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:35.392 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.400+0000 7f2f51d9b700 1 --2- 172.21.15.23:0/1654348015 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2f3c05c2b0 0x7f2f3c05e770 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f2f54106b20 tx=0x7f2f48009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:35.392 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.404+0000 7f2f42ffd700 1 -- 172.21.15.23:0/1654348015 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f44048ed0 con 0x7f2f540f5660 2021-11-19T08:44:35.587 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.603+0000 7f2f5359e700 1 -- 172.21.15.23:0/1654348015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f54045b50 con 0x7f2f540f5660 2021-11-19T08:44:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.603+0000 7f2f42ffd700 1 -- 172.21.15.23:0/1654348015 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f2f54045b50 con 0x7f2f540f5660 2021-11-19T08:44:35.588 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:35.588 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:27.239456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:35.590 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.605+0000 7f2f5359e700 1 -- 172.21.15.23:0/1654348015 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2f3c05c2b0 msgr2=0x7f2f3c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:35.590 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.605+0000 7f2f5359e700 1 --2- 172.21.15.23:0/1654348015 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2f3c05c2b0 0x7f2f3c05e770 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f2f54106b20 tx=0x7f2f48009b30).stop 2021-11-19T08:44:35.590 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.605+0000 7f2f5359e700 1 -- 172.21.15.23:0/1654348015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f540f5660 msgr2=0x7f2f54106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:35.590 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.605+0000 7f2f5359e700 1 --2- 172.21.15.23:0/1654348015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f540f5660 0x7f2f54106d90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f2f44002700 tx=0x7f2f44009d80).stop 2021-11-19T08:44:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.605+0000 7f2f5359e700 1 -- 172.21.15.23:0/1654348015 shutdown_connections 2021-11-19T08:44:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.605+0000 7f2f5359e700 1 --2- 172.21.15.23:0/1654348015 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2f3c05c2b0 0x7f2f3c05e770 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.605+0000 7f2f5359e700 1 --2- 172.21.15.23:0/1654348015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f540f5660 0x7f2f54106d90 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.605+0000 7f2f5359e700 1 --2- 172.21.15.23:0/1654348015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f540f6960 0x7f2f54107280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.605+0000 7f2f5359e700 1 -- 172.21.15.23:0/1654348015 >> 172.21.15.23:0/1654348015 conn(0x7f2f540f0230 msgr2=0x7f2f540f4550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.605+0000 7f2f5359e700 1 -- 172.21.15.23:0/1654348015 shutdown_connections 2021-11-19T08:44:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:35.605+0000 7f2f5359e700 1 -- 172.21.15.23:0/1654348015 wait complete. 2021-11-19T08:44:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 83 2021-11-19T08:44:35.601 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:27.239456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:35.601 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:rejoin 2021-11-19T08:44:36.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:36 smithi134 conmon[36363]: cluster 2021-11-19T08:44:34.892231+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:44:36.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:36 smithi134 conmon[36363]: 14164) 317 : cluster [DBG] pgmap v292: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 987 B/s rd, 0 op/s 2021-11-19T08:44:36.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:36 smithi134 conmon[36363]: audit 2021-11-19T08:44:35.604218+0000 mon.smithi023 (mon. 2021-11-19T08:44:36.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:36 smithi134 conmon[36363]: 0) 850 : audit [DBG] from='client.? 172.21.15.23:0/1654348015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:36.602 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:36.631 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:36 smithi023 conmon[33490]: cluster 2021-11-19T08:44:34.892231+0000 mgr.smithi023.ehlpcl (mgr.14164) 317 : cluster [DBG] pgmap v292: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 987 B/s rd, 0 op/s 2021-11-19T08:44:36.631 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:36 smithi023 conmon[33490]: audit 2021-11-19T08:44:35.604218+0000 mon.smithi023 (mon.0) 850 : audit [DBG] from='client.? 172.21.15.23:0/1654348015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:36.737 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.750+0000 7f8dda11f700 1 -- 172.21.15.23:0/1882578083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8dd40f7940 msgr2=0x7f8dd40f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:36.737 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.750+0000 7f8dda11f700 1 --2- 172.21.15.23:0/1882578083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8dd40f7940 0x7f8dd40f7da0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f8dc8004660 tx=0x7f8dc8009b30).stop 2021-11-19T08:44:36.737 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.750+0000 7f8dda11f700 1 -- 172.21.15.23:0/1882578083 shutdown_connections 2021-11-19T08:44:36.738 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.750+0000 7f8dda11f700 1 --2- 172.21.15.23:0/1882578083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8dd40f7940 0x7f8dd40f7da0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:36.738 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.750+0000 7f8dda11f700 1 --2- 172.21.15.23:0/1882578083 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dd40f66a0 0x7f8dd40f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:36.738 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.750+0000 7f8dda11f700 1 -- 172.21.15.23:0/1882578083 >> 172.21.15.23:0/1882578083 conn(0x7f8dd40f1da0 msgr2=0x7f8dd40f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:36.739 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.750+0000 7f8dda11f700 1 -- 172.21.15.23:0/1882578083 shutdown_connections 2021-11-19T08:44:36.739 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.750+0000 7f8dda11f700 1 -- 172.21.15.23:0/1882578083 wait complete. 2021-11-19T08:44:36.739 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.750+0000 7f8dda11f700 1 Processor -- start 2021-11-19T08:44:36.739 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.750+0000 7f8dda11f700 1 -- start start 2021-11-19T08:44:36.740 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.751+0000 7f8dda11f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dd40f66a0 0x7f8dd406b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:36.740 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.751+0000 7f8dda11f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8dd40f7940 0x7f8dd4069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:36.740 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.751+0000 7f8dda11f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd406bd70 con 0x7f8dd40f7940 2021-11-19T08:44:36.740 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.751+0000 7f8dda11f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd406a310 con 0x7f8dd40f66a0 2021-11-19T08:44:36.741 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.751+0000 7f8dd2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8dd40f7940 0x7f8dd4069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:36.741 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.751+0000 7f8dd2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8dd40f7940 0x7f8dd4069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48518/0 (socket says 172.21.15.23:48518) 2021-11-19T08:44:36.741 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.751+0000 7f8dd2ffd700 1 -- 172.21.15.23:0/3786522094 learned_addr learned my addr 172.21.15.23:0/3786522094 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:36.741 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.751+0000 7f8dd37fe700 1 --2- 172.21.15.23:0/3786522094 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dd40f66a0 0x7f8dd406b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:36.742 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.751+0000 7f8dd2ffd700 1 -- 172.21.15.23:0/3786522094 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dd40f66a0 msgr2=0x7f8dd406b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:36.742 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.751+0000 7f8dd2ffd700 1 --2- 172.21.15.23:0/3786522094 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dd40f66a0 0x7f8dd406b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:36.742 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.751+0000 7f8dd2ffd700 1 -- 172.21.15.23:0/3786522094 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc8005040 con 0x7f8dd40f7940 2021-11-19T08:44:36.742 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.751+0000 7f8dd37fe700 1 --2- 172.21.15.23:0/3786522094 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dd40f66a0 0x7f8dd406b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:44:36.742 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.751+0000 7f8dd2ffd700 1 --2- 172.21.15.23:0/3786522094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8dd40f7940 0x7f8dd4069e20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f8dc8009830 tx=0x7f8dc8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:36.743 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.751+0000 7f8dbbfff700 1 -- 172.21.15.23:0/3786522094 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dc8018070 con 0x7f8dd40f7940 2021-11-19T08:44:36.743 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.752+0000 7f8dda11f700 1 -- 172.21.15.23:0/3786522094 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd406a530 con 0x7f8dd40f7940 2021-11-19T08:44:36.743 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.752+0000 7f8dbbfff700 1 -- 172.21.15.23:0/3786522094 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8dc8013410 con 0x7f8dd40f7940 2021-11-19T08:44:36.743 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.752+0000 7f8dbbfff700 1 -- 172.21.15.23:0/3786522094 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dc801c620 con 0x7f8dd40f7940 2021-11-19T08:44:36.744 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.752+0000 7f8dda11f700 1 -- 172.21.15.23:0/3786522094 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd406aa90 con 0x7f8dd40f7940 2021-11-19T08:44:36.744 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.752+0000 7f8dda11f700 1 -- 172.21.15.23:0/3786522094 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd4046330 con 0x7f8dd40f7940 2021-11-19T08:44:36.745 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.753+0000 7f8dbbfff700 1 -- 172.21.15.23:0/3786522094 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8dc801c8e0 con 0x7f8dd40f7940 2021-11-19T08:44:36.745 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.753+0000 7f8dbbfff700 1 --2- 172.21.15.23:0/3786522094 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8dbc05c2a0 0x7f8dbc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:36.746 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.753+0000 7f8dbbfff700 1 -- 172.21.15.23:0/3786522094 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f8dc8078270 con 0x7f8dd40f7940 2021-11-19T08:44:36.746 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.753+0000 7f8dd37fe700 1 --2- 172.21.15.23:0/3786522094 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8dbc05c2a0 0x7f8dbc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:36.746 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.754+0000 7f8dd37fe700 1 --2- 172.21.15.23:0/3786522094 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8dbc05c2a0 0x7f8dbc05e760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f8dd40f77a0 tx=0x7f8dcc009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:36.746 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.757+0000 7f8dbbfff700 1 -- 172.21.15.23:0/3786522094 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8dc804aa20 con 0x7f8dd40f7940 2021-11-19T08:44:36.938 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.953+0000 7f8dda11f700 1 -- 172.21.15.23:0/3786522094 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dd4027a00 con 0x7f8dd40f7940 2021-11-19T08:44:36.938 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.954+0000 7f8dbbfff700 1 -- 172.21.15.23:0/3786522094 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f8dc804e040 con 0x7f8dd40f7940 2021-11-19T08:44:36.939 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:36.939 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:27.239456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:36.941 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.956+0000 7f8dda11f700 1 -- 172.21.15.23:0/3786522094 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8dbc05c2a0 msgr2=0x7f8dbc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:36.941 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.956+0000 7f8dda11f700 1 --2- 172.21.15.23:0/3786522094 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8dbc05c2a0 0x7f8dbc05e760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f8dd40f77a0 tx=0x7f8dcc009040).stop 2021-11-19T08:44:36.942 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.956+0000 7f8dda11f700 1 -- 172.21.15.23:0/3786522094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8dd40f7940 msgr2=0x7f8dd4069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:36.942 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.956+0000 7f8dda11f700 1 --2- 172.21.15.23:0/3786522094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8dd40f7940 0x7f8dd4069e20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f8dc8009830 tx=0x7f8dc8009b30).stop 2021-11-19T08:44:36.943 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.956+0000 7f8dda11f700 1 -- 172.21.15.23:0/3786522094 shutdown_connections 2021-11-19T08:44:36.943 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.956+0000 7f8dda11f700 1 --2- 172.21.15.23:0/3786522094 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8dd40f66a0 0x7f8dd406b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:36.943 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.956+0000 7f8dda11f700 1 --2- 172.21.15.23:0/3786522094 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8dbc05c2a0 0x7f8dbc05e760 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:36.944 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.956+0000 7f8dda11f700 1 --2- 172.21.15.23:0/3786522094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8dd40f7940 0x7f8dd4069e20 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:36.944 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.956+0000 7f8dda11f700 1 -- 172.21.15.23:0/3786522094 >> 172.21.15.23:0/3786522094 conn(0x7f8dd40f1da0 msgr2=0x7f8dd40fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:36.945 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.956+0000 7f8dda11f700 1 -- 172.21.15.23:0/3786522094 shutdown_connections 2021-11-19T08:44:36.945 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:36.956+0000 7f8dda11f700 1 -- 172.21.15.23:0/3786522094 wait complete. 2021-11-19T08:44:36.945 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 83 2021-11-19T08:44:36.953 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:27.239456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:36.954 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:rejoin 2021-11-19T08:44:37.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:37 smithi023 conmon[33490]: audit 2021-11-19T08:44:36.954838+0000 mon.smithi023 (mon.0) 851 : audit [DBG] from='client.? 172.21.15.23:0/3786522094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:37.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:37 smithi134 conmon[36363]: audit 2021-11-19T08:44:36.954838+0000 mon.smithi023 (mon.0) 851 : 2021-11-19T08:44:37.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:37 smithi134 conmon[36363]: audit [DBG] from='client.? 172.21.15.23:0/3786522094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:37.956 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:38.083 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.095+0000 7f2a30af2700 1 -- 172.21.15.23:0/2099172497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a2c0f5460 msgr2=0x7f2a2c0f5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:38.083 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.095+0000 7f2a30af2700 1 --2- 172.21.15.23:0/2099172497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a2c0f5460 0x7f2a2c0f5880 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f2a20004660 tx=0x7f2a20009b30).stop 2021-11-19T08:44:38.083 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.095+0000 7f2a30af2700 1 -- 172.21.15.23:0/2099172497 shutdown_connections 2021-11-19T08:44:38.083 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.095+0000 7f2a30af2700 1 --2- 172.21.15.23:0/2099172497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c0f6700 0x7f2a2c0f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:38.084 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.095+0000 7f2a30af2700 1 --2- 172.21.15.23:0/2099172497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a2c0f5460 0x7f2a2c0f5880 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:38.084 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.095+0000 7f2a30af2700 1 -- 172.21.15.23:0/2099172497 >> 172.21.15.23:0/2099172497 conn(0x7f2a2c0f0ba0 msgr2=0x7f2a2c0f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:38.084 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.095+0000 7f2a30af2700 1 -- 172.21.15.23:0/2099172497 shutdown_connections 2021-11-19T08:44:38.084 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.095+0000 7f2a30af2700 1 -- 172.21.15.23:0/2099172497 wait complete. 2021-11-19T08:44:38.084 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.096+0000 7f2a30af2700 1 Processor -- start 2021-11-19T08:44:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.096+0000 7f2a30af2700 1 -- start start 2021-11-19T08:44:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.096+0000 7f2a30af2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c0f5460 0x7f2a2c109a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:38.086 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.096+0000 7f2a30af2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a2c0f6700 0x7f2a2c10bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:38.086 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.096+0000 7f2a30af2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c0601f0 con 0x7f2a2c0f6700 2021-11-19T08:44:38.087 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.096+0000 7f2a30af2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c060330 con 0x7f2a2c0f5460 2021-11-19T08:44:38.087 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.096+0000 7f2a2affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a2c0f6700 0x7f2a2c10bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:38.088 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.096+0000 7f2a2affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a2c0f6700 0x7f2a2c10bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48526/0 (socket says 172.21.15.23:48526) 2021-11-19T08:44:38.088 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.096+0000 7f2a2affd700 1 -- 172.21.15.23:0/3493910547 learned_addr learned my addr 172.21.15.23:0/3493910547 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:38.089 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.096+0000 7f2a2b7fe700 1 --2- 172.21.15.23:0/3493910547 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c0f5460 0x7f2a2c109a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:38.089 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.096+0000 7f2a2affd700 1 -- 172.21.15.23:0/3493910547 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c0f5460 msgr2=0x7f2a2c109a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.096+0000 7f2a2affd700 1 --2- 172.21.15.23:0/3493910547 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c0f5460 0x7f2a2c109a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.096+0000 7f2a2affd700 1 -- 172.21.15.23:0/3493910547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a20005040 con 0x7f2a2c0f6700 2021-11-19T08:44:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.096+0000 7f2a2affd700 1 --2- 172.21.15.23:0/3493910547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a2c0f6700 0x7f2a2c10bf00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f2a1c002700 tx=0x7f2a1c009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.097+0000 7f2a13fff700 1 -- 172.21.15.23:0/3493910547 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a1c009320 con 0x7f2a2c0f6700 2021-11-19T08:44:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.097+0000 7f2a13fff700 1 -- 172.21.15.23:0/3493910547 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a1c00f040 con 0x7f2a2c0f6700 2021-11-19T08:44:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.097+0000 7f2a30af2700 1 -- 172.21.15.23:0/3493910547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2c10c4b0 con 0x7f2a2c0f6700 2021-11-19T08:44:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.097+0000 7f2a13fff700 1 -- 172.21.15.23:0/3493910547 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a1c014820 con 0x7f2a2c0f6700 2021-11-19T08:44:38.093 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.097+0000 7f2a30af2700 1 -- 172.21.15.23:0/3493910547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2c10ca40 con 0x7f2a2c0f6700 2021-11-19T08:44:38.093 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.098+0000 7f2a30af2700 1 -- 172.21.15.23:0/3493910547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2c045ad0 con 0x7f2a2c0f6700 2021-11-19T08:44:38.095 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.099+0000 7f2a13fff700 1 -- 172.21.15.23:0/3493910547 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2a1c0077b0 con 0x7f2a2c0f6700 2021-11-19T08:44:38.095 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.099+0000 7f2a13fff700 1 --2- 172.21.15.23:0/3493910547 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2a1405c300 0x7f2a1405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:38.096 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.099+0000 7f2a13fff700 1 -- 172.21.15.23:0/3493910547 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6586+0+0 (secure 0 0 0) 0x7f2a1c0768f0 con 0x7f2a2c0f6700 2021-11-19T08:44:38.096 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.099+0000 7f2a2b7fe700 1 --2- 172.21.15.23:0/3493910547 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2a1405c300 0x7f2a1405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:38.097 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.100+0000 7f2a2b7fe700 1 --2- 172.21.15.23:0/3493910547 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2a1405c300 0x7f2a1405e7c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f2a2c05fe70 tx=0x7f2a20009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:38.098 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.102+0000 7f2a13fff700 1 -- 172.21.15.23:0/3493910547 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a1c049180 con 0x7f2a2c0f6700 2021-11-19T08:44:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.300+0000 7f2a30af2700 1 -- 172.21.15.23:0/3493910547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a2c027a00 con 0x7f2a2c0f6700 2021-11-19T08:44:38.286 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.301+0000 7f2a13fff700 1 -- 172.21.15.23:0/3493910547 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f2a1c04c7a0 con 0x7f2a2c0f6700 2021-11-19T08:44:38.287 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:38.287 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":83},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":82,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:27.239456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14676,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14676":{"gid":14676,"name":"cephfs.smithi023.wazanl","rank":0,"incarnation":76,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.23:6829/3188238627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3188238627},{"type":"v1","addr":"172.21.15.23:6829","nonce":3188238627}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.302+0000 7f2a30af2700 1 -- 172.21.15.23:0/3493910547 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2a1405c300 msgr2=0x7f2a1405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.302+0000 7f2a30af2700 1 --2- 172.21.15.23:0/3493910547 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2a1405c300 0x7f2a1405e7c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f2a2c05fe70 tx=0x7f2a20009b30).stop 2021-11-19T08:44:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.302+0000 7f2a30af2700 1 -- 172.21.15.23:0/3493910547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a2c0f6700 msgr2=0x7f2a2c10bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.302+0000 7f2a30af2700 1 --2- 172.21.15.23:0/3493910547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a2c0f6700 0x7f2a2c10bf00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f2a1c002700 tx=0x7f2a1c009e00).stop 2021-11-19T08:44:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.302+0000 7f2a30af2700 1 -- 172.21.15.23:0/3493910547 shutdown_connections 2021-11-19T08:44:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.302+0000 7f2a30af2700 1 --2- 172.21.15.23:0/3493910547 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c0f5460 0x7f2a2c109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.302+0000 7f2a30af2700 1 --2- 172.21.15.23:0/3493910547 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2a1405c300 0x7f2a1405e7c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.302+0000 7f2a30af2700 1 --2- 172.21.15.23:0/3493910547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a2c0f6700 0x7f2a2c10bf00 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.302+0000 7f2a30af2700 1 -- 172.21.15.23:0/3493910547 >> 172.21.15.23:0/3493910547 conn(0x7f2a2c0f0ba0 msgr2=0x7f2a2c0fa150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.302+0000 7f2a30af2700 1 -- 172.21.15.23:0/3493910547 shutdown_connections 2021-11-19T08:44:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:38.303+0000 7f2a30af2700 1 -- 172.21.15.23:0/3493910547 wait complete. 2021-11-19T08:44:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 83 2021-11-19T08:44:38.299 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:27.239456+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 55, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14676, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14676': {'gid': 14676, 'name': 'cephfs.smithi023.wazanl', 'rank': 0, 'incarnation': 76, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.23:6829/3188238627', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3188238627}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3188238627}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:38.300 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14676:up:rejoin 2021-11-19T08:44:38.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:38 smithi023 conmon[33490]: cluster 2021-11-19T08:44:36.892633+0000 mgr.smithi023.ehlpcl (mgr.14164) 318 : cluster [DBG] pgmap v293: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 341 B/s rd, 0 op/s 2021-11-19T08:44:38.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:38 smithi023 conmon[33490]: cluster 2021-11-19T08:44:38.187284+0000 mon.smithi023 (mon.0) 852 : cluster [INF] daemon mds.cephfs.smithi023.wazanl restarted 2021-11-19T08:44:38.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:38 smithi023 conmon[33490]: cluster 2021-11-19T08:44:38 2021-11-19T08:44:38.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:38 smithi023 conmon[33490]: .191629+0000 mon.smithi023 (mon.0) 853 : cluster [DBG] osdmap e56: 6 total, 6 up, 6 in 2021-11-19T08:44:38.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:38 smithi023 conmon[33490]: audit 2021-11-19T08 2021-11-19T08:44:38.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:38 smithi023 conmon[33490]: :44:38.301677+0000 mon.smithi023 (mon.0) 854 2021-11-19T08:44:38.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:38 smithi023 conmon[33490]: : audit [DBG] from='client.? 172.21.15.23:0/3493910547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:38.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:38 smithi134 conmon[36363]: cluster 2021-11-19T08:44:36.892633+0000 mgr.smithi023.ehlpcl (mgr.14164) 318 : 2021-11-19T08:44:38.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:38 smithi134 conmon[36363]: cluster [DBG] pgmap v293: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 341 B/s rd, 0 op/s 2021-11-19T08:44:38.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:38 smithi134 conmon[36363]: cluster 2021-11-19T08:44:38.187284+0000 mon.smithi023 (mon.0) 852 : cluster [INF] daemon mds.cephfs.smithi023.wazanl restarted 2021-11-19T08:44:38.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:38 smithi134 conmon[36363]: cluster 2021-11-19T 2021-11-19T08:44:38.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:38 smithi134 conmon[36363]: 08:44:38.191629+0000 mon.smithi023 (mon.0) 853 : cluster [DBG] osdmap e56: 6 total, 6 up, 6 in 2021-11-19T08:44:38.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:38 smithi134 conmon[36363]: audit 2021-11-19T08:44:38.301677 2021-11-19T08:44:38.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:38 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 854 : audit [DBG] from='client.? 172.21.15.23:0/3493910547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:39.302 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:39.437 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.449+0000 7f02c759e700 1 -- 172.21.15.23:0/474890881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02c80f6290 msgr2=0x7f02c80f66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:39.437 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.449+0000 7f02c759e700 1 --2- 172.21.15.23:0/474890881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02c80f6290 0x7f02c80f66f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f02bc004660 tx=0x7f02bc009b30).stop 2021-11-19T08:44:39.437 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.450+0000 7f02c759e700 1 -- 172.21.15.23:0/474890881 shutdown_connections 2021-11-19T08:44:39.438 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.450+0000 7f02c759e700 1 --2- 172.21.15.23:0/474890881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02c80f6290 0x7f02c80f66f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:39.438 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.450+0000 7f02c759e700 1 --2- 172.21.15.23:0/474890881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02c80f7c80 0x7f02c80f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:39.438 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.450+0000 7f02c759e700 1 -- 172.21.15.23:0/474890881 >> 172.21.15.23:0/474890881 conn(0x7f02c80f1d10 msgr2=0x7f02c80f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:39.438 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.450+0000 7f02c759e700 1 -- 172.21.15.23:0/474890881 shutdown_connections 2021-11-19T08:44:39.438 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.450+0000 7f02c759e700 1 -- 172.21.15.23:0/474890881 wait complete. 2021-11-19T08:44:39.439 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.450+0000 7f02c759e700 1 Processor -- start 2021-11-19T08:44:39.439 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.450+0000 7f02c759e700 1 -- start start 2021-11-19T08:44:39.439 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.450+0000 7f02c759e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02c80f6290 0x7f02c8106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:39.440 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.450+0000 7f02c759e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02c80f7c80 0x7f02c8107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:39.440 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.450+0000 7f02c759e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c8107940 con 0x7f02c80f7c80 2021-11-19T08:44:39.440 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.450+0000 7f02c759e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c8108140 con 0x7f02c80f6290 2021-11-19T08:44:39.440 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.450+0000 7f02c5d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02c80f7c80 0x7f02c8107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:39.441 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.451+0000 7f02c5d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02c80f7c80 0x7f02c8107390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48558/0 (socket says 172.21.15.23:48558) 2021-11-19T08:44:39.441 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.451+0000 7f02c5d9b700 1 -- 172.21.15.23:0/3509584293 learned_addr learned my addr 172.21.15.23:0/3509584293 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:39.441 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.451+0000 7f02c659c700 1 --2- 172.21.15.23:0/3509584293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02c80f6290 0x7f02c8106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:39.441 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.451+0000 7f02c5d9b700 1 -- 172.21.15.23:0/3509584293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02c80f6290 msgr2=0x7f02c8106ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:39.441 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.451+0000 7f02c5d9b700 1 --2- 172.21.15.23:0/3509584293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02c80f6290 0x7f02c8106ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.451+0000 7f02c5d9b700 1 -- 172.21.15.23:0/3509584293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02bc005040 con 0x7f02c80f7c80 2021-11-19T08:44:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.451+0000 7f02c5d9b700 1 --2- 172.21.15.23:0/3509584293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02c80f7c80 0x7f02c8107390 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f02bc009f50 tx=0x7f02bc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.451+0000 7f02b6ffd700 1 -- 172.21.15.23:0/3509584293 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02bc018070 con 0x7f02c80f7c80 2021-11-19T08:44:39.442 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.451+0000 7f02b6ffd700 1 -- 172.21.15.23:0/3509584293 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02bc013410 con 0x7f02c80f7c80 2021-11-19T08:44:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.451+0000 7f02b6ffd700 1 -- 172.21.15.23:0/3509584293 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02bc01c3c0 con 0x7f02c80f7c80 2021-11-19T08:44:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.451+0000 7f02c759e700 1 -- 172.21.15.23:0/3509584293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c81083c0 con 0x7f02c80f7c80 2021-11-19T08:44:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.451+0000 7f02c759e700 1 -- 172.21.15.23:0/3509584293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c8108920 con 0x7f02c80f7c80 2021-11-19T08:44:39.443 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.452+0000 7f02c759e700 1 -- 172.21.15.23:0/3509584293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c80f59c0 con 0x7f02c80f7c80 2021-11-19T08:44:39.444 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.453+0000 7f02b6ffd700 1 -- 172.21.15.23:0/3509584293 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f02bc01d420 con 0x7f02c80f7c80 2021-11-19T08:44:39.445 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.453+0000 7f02b6ffd700 1 --2- 172.21.15.23:0/3509584293 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f02b005c300 0x7f02b005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:39.445 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.453+0000 7f02b6ffd700 1 -- 172.21.15.23:0/3509584293 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f02bc078040 con 0x7f02c80f7c80 2021-11-19T08:44:39.445 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.453+0000 7f02c659c700 1 --2- 172.21.15.23:0/3509584293 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f02b005c300 0x7f02b005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:39.445 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.453+0000 7f02c659c700 1 --2- 172.21.15.23:0/3509584293 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f02b005c300 0x7f02b005e7c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f02c80f76d0 tx=0x7f02b800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:39.446 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.457+0000 7f02b6ffd700 1 -- 172.21.15.23:0/3509584293 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02bc04a800 con 0x7f02c80f7c80 2021-11-19T08:44:39.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:39 smithi134 conmon[36363]: cluster 2021-11-19T08:44:38.406385+0000 mon.smithi023 (mon. 2021-11-19T08:44:39.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:39 smithi134 conmon[36363]: 0) 855 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3287241569,v1:172.21.15.23:6829/3287241569] up:boot 2021-11-19T08:44:39.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:39 smithi134 conmon[36363]: cluster 2021-11-19T08:44:38.406443+0000 mon.smithi023 (mon.0) 856 : cluster [INF] Standby daemon mds.cephfs.smithi023.fuvzkr assigned to filesystem cephfs as rank 0 2021-11-19T08:44:39.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:39 smithi134 conmon[36363]: 2021-11-19T08:44:39.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:39 smithi134 conmon[36363]: cluster 2021-11-19 2021-11-19T08:44:39.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:39 smithi134 conmon[36363]: T08:44:38.406896+0000 mon.smithi023 (mon.0) 857 : cluster 2021-11-19T08:44:39.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:39 smithi134 conmon[36363]: [DBG] fsmap cephfs:1/2 {1=cephfs.smithi134.opxllb=up:rejoin} 3 up:standby, 1 failed 2021-11-19T08:44:39.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:39 smithi134 conmon[36363]: audit 2021-11-19T08:44: 2021-11-19T08:44:39.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:39 smithi134 conmon[36363]: 38.407023+0000 mon.smithi023 (mon.0) 858 : audit 2021-11-19T08:44:39.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:39 smithi134 conmon[36363]: [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.wazanl"}]: dispatch 2021-11-19T08:44:39.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:39 smithi134 conmon[36363]: cluster 2021-11-19T08: 2021-11-19T08:44:39.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:39 smithi134 conmon[36363]: 44:38.412339+0000 mon.smithi023 (mon.0) 2021-11-19T08:44:39.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:39 smithi134 conmon[36363]: 859 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:replay,1=cephfs.smithi134.opxllb=up:rejoin} 2 up:standby 2021-11-19T08:44:39.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: cluster 2021-11-19T08:44:38 2021-11-19T08:44:39.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: .406385+0000 mon.smithi023 (mon.0) 855 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3287241569,v1:172.21.15.23:6829/3287241569] up:boot 2021-11-19T08:44:39.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: cluster 2021-11-19T08:44:39.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: 2021-11-19T08:44:38.406443+0000 2021-11-19T08:44:39.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: mon.smithi023 (mon.0) 856 2021-11-19T08:44:39.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: : cluster [INF] Standby daemon mds.cephfs.smithi023.fuvzkr assigned to filesystem cephfs as rank 0 2021-11-19T08:44:39.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: cluster 2021 2021-11-19T08:44:39.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: -11-19T08:44:38 2021-11-19T08:44:39.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: .406896+0000 mon.smithi023 (mon. 2021-11-19T08:44:39.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: 0) 857 : cluster [DBG] 2021-11-19T08:44:39.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: fsmap cephfs:1/2 {1=cephfs.smithi134.opxllb=up:rejoin} 3 up:standby, 1 failed 2021-11-19T08:44:39.646 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: audit 2021 2021-11-19T08:44:39.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: -11-19T08:44: 2021-11-19T08:44:39.647 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: 38.407023+0000 mon.smithi023 (mon 2021-11-19T08:44:39.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: .0) 858 : audit [DBG] 2021-11-19T08:44:39.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi023.wazanl"}]: dispatch 2021-11-19T08:44:39.648 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: cluster 2021 2021-11-19T08:44:39.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: -11-19T08:44: 2021-11-19T08:44:39.649 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: 38.412339+0000 mon.smithi023 (mon 2021-11-19T08:44:39.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: .0) 859 : cluster [DBG] 2021-11-19T08:44:39.650 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:39 smithi023 conmon[33490]: fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:replay,1=cephfs.smithi134.opxllb=up:rejoin} 2 up:standby 2021-11-19T08:44:39.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.658+0000 7f02c759e700 1 -- 172.21.15.23:0/3509584293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02c8045ad0 con 0x7f02c80f7c80 2021-11-19T08:44:39.652 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.659+0000 7f02b6ffd700 1 -- 172.21.15.23:0/3509584293 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2953 (secure 0 0 0) 0x7f02c8045ad0 con 0x7f02c80f7c80 2021-11-19T08:44:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:39.654 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":84},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:39.410151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:39.658 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.661+0000 7f02c759e700 1 -- 172.21.15.23:0/3509584293 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f02b005c300 msgr2=0x7f02b005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:39.658 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.661+0000 7f02c759e700 1 --2- 172.21.15.23:0/3509584293 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f02b005c300 0x7f02b005e7c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f02c80f76d0 tx=0x7f02b800b040).stop 2021-11-19T08:44:39.659 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.661+0000 7f02c759e700 1 -- 172.21.15.23:0/3509584293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02c80f7c80 msgr2=0x7f02c8107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:39.659 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.661+0000 7f02c759e700 1 --2- 172.21.15.23:0/3509584293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02c80f7c80 0x7f02c8107390 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f02bc009f50 tx=0x7f02bc009b30).stop 2021-11-19T08:44:39.659 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.661+0000 7f02c759e700 1 -- 172.21.15.23:0/3509584293 shutdown_connections 2021-11-19T08:44:39.659 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.661+0000 7f02c759e700 1 --2- 172.21.15.23:0/3509584293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02c80f6290 0x7f02c8106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:39.660 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.661+0000 7f02c759e700 1 --2- 172.21.15.23:0/3509584293 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f02b005c300 0x7f02b005e7c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:39.660 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.661+0000 7f02c759e700 1 --2- 172.21.15.23:0/3509584293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02c80f7c80 0x7f02c8107390 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:39.660 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.661+0000 7f02c759e700 1 -- 172.21.15.23:0/3509584293 >> 172.21.15.23:0/3509584293 conn(0x7f02c80f1d10 msgr2=0x7f02c80fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:39.660 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.661+0000 7f02c759e700 1 -- 172.21.15.23:0/3509584293 shutdown_connections 2021-11-19T08:44:39.660 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:39.661+0000 7f02c759e700 1 -- 172.21.15.23:0/3509584293 wait complete. 2021-11-19T08:44:39.661 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 86 2021-11-19T08:44:39.661 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:39.410151+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:39.662 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:resolve 2021-11-19T08:44:40.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:40 smithi023 conmon[33490]: cluster 2021-11-19T08:44:38.893315+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:44:40.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:40 smithi023 conmon[33490]: ) 319 : cluster [DBG] pgmap v295: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:44:40.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:40 smithi023 conmon[33490]: cluster 2021-11-19T08:44:39.419341+0000 mon.smithi023 (mon.0) 860 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3968768531,v1:172.21.15.23:6827/3968768531] up:resolve 2021-11-19T08:44:40.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:40 smithi023 conmon[33490]: cluster 2021-11-19T08:44:39.419418+0000 mon.smithi023 (mon.0) 861 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi134.opxllb=up:rejoin} 2 up:standby 2021-11-19T08:44:40.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:40 smithi023 conmon[33490]: audit 2021-11-19T08:44:40.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:40 smithi023 conmon[33490]: 2021-11-19T08:44:39.659953+0000 mon.smithi023 (mon.0) 862 : audit [DBG] from='client.? 172.21.15.23:0/3509584293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:40.663 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:40.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:40 smithi134 conmon[36363]: cluster 2021-11-19T08:44:38.893315+0000 mgr.smithi023.ehlpcl (mgr.14164) 319 : 2021-11-19T08:44:40.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:40 smithi134 conmon[36363]: cluster [DBG] pgmap v295: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:44:40.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:40 smithi134 conmon[36363]: cluster 2021-11-19T08:44:39.419341+0000 mon.smithi023 (mon.0) 860 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3968768531,v1:172.21.15.23:6827/3968768531] up:resolve 2021-11-19T08:44:40.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:40 smithi134 conmon[36363]: cluster 2021-11-19T08:44:39.419418+0000 mon.smithi023 ( 2021-11-19T08:44:40.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:40 smithi134 conmon[36363]: mon.0) 861 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi134.opxllb=up:rejoin} 2 up:standby 2021-11-19T08:44:40.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:40 smithi134 conmon[36363]: audit 2021- 2021-11-19T08:44:40.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:40 smithi134 conmon[36363]: 11-19T08:44:39.659953+0000 mon.smithi023 (mon. 2021-11-19T08:44:40.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:40 smithi134 conmon[36363]: 0) 862 : audit [DBG] from='client.? 172.21.15.23:0/3509584293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:40.795 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.811+0000 7f74b2039700 1 -- 172.21.15.23:0/2599684164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74ac0f6820 msgr2=0x7f74ac0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:40.795 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.811+0000 7f74b2039700 1 --2- 172.21.15.23:0/2599684164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74ac0f6820 0x7f74ac0f6c80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f74a0004660 tx=0x7f74a0009b30).stop 2021-11-19T08:44:40.796 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.811+0000 7f74b2039700 1 -- 172.21.15.23:0/2599684164 shutdown_connections 2021-11-19T08:44:40.796 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.811+0000 7f74b2039700 1 --2- 172.21.15.23:0/2599684164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74ac0f6820 0x7f74ac0f6c80 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.811+0000 7f74b2039700 1 --2- 172.21.15.23:0/2599684164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74ac0f5580 0x7f74ac0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.811+0000 7f74b2039700 1 -- 172.21.15.23:0/2599684164 >> 172.21.15.23:0/2599684164 conn(0x7f74ac0f0ce0 msgr2=0x7f74ac0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.811+0000 7f74b2039700 1 -- 172.21.15.23:0/2599684164 shutdown_connections 2021-11-19T08:44:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.811+0000 7f74b2039700 1 -- 172.21.15.23:0/2599684164 wait complete. 2021-11-19T08:44:40.800 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.811+0000 7f74b2039700 1 Processor -- start 2021-11-19T08:44:40.800 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.812+0000 7f74b2039700 1 -- start start 2021-11-19T08:44:40.800 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.812+0000 7f74b2039700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74ac0f5580 0x7f74ac109030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:40.801 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.812+0000 7f74b2039700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74ac0f6820 0x7f74ac109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:40.801 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.812+0000 7f74b2039700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac109ad0 con 0x7f74ac0f6820 2021-11-19T08:44:40.802 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.812+0000 7f74b2039700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac109c10 con 0x7f74ac0f5580 2021-11-19T08:44:40.802 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.812+0000 7f74aaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74ac0f6820 0x7f74ac109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:40.802 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.812+0000 7f74aaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74ac0f6820 0x7f74ac109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48578/0 (socket says 172.21.15.23:48578) 2021-11-19T08:44:40.802 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.812+0000 7f74aaffd700 1 -- 172.21.15.23:0/444391905 learned_addr learned my addr 172.21.15.23:0/444391905 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:40.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.812+0000 7f74ab7fe700 1 --2- 172.21.15.23:0/444391905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74ac0f5580 0x7f74ac109030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:40.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.812+0000 7f74aaffd700 1 -- 172.21.15.23:0/444391905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74ac0f5580 msgr2=0x7f74ac109030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:40.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.812+0000 7f74aaffd700 1 --2- 172.21.15.23:0/444391905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74ac0f5580 0x7f74ac109030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:40.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.812+0000 7f74aaffd700 1 -- 172.21.15.23:0/444391905 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a0005040 con 0x7f74ac0f6820 2021-11-19T08:44:40.803 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.812+0000 7f74aaffd700 1 --2- 172.21.15.23:0/444391905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74ac0f6820 0x7f74ac109520 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f74a0000c00 tx=0x7f74a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:40.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.813+0000 7f7493fff700 1 -- 172.21.15.23:0/444391905 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74a0019070 con 0x7f74ac0f6820 2021-11-19T08:44:40.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.813+0000 7f7493fff700 1 -- 172.21.15.23:0/444391905 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74a0007e70 con 0x7f74ac0f6820 2021-11-19T08:44:40.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.813+0000 7f74b2039700 1 -- 172.21.15.23:0/444391905 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ac10e650 con 0x7f74ac0f6820 2021-11-19T08:44:40.804 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.813+0000 7f7493fff700 1 -- 172.21.15.23:0/444391905 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74a0014d00 con 0x7f74ac0f6820 2021-11-19T08:44:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.813+0000 7f74b2039700 1 -- 172.21.15.23:0/444391905 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74ac10ec10 con 0x7f74ac0f6820 2021-11-19T08:44:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.814+0000 7f74b2039700 1 -- 172.21.15.23:0/444391905 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ac0fdd10 con 0x7f74ac0f6820 2021-11-19T08:44:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.814+0000 7f7493fff700 1 -- 172.21.15.23:0/444391905 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f74a0025070 con 0x7f74ac0f6820 2021-11-19T08:44:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.814+0000 7f7493fff700 1 --2- 172.21.15.23:0/444391905 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f749405c2a0 0x7f749405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.814+0000 7f7493fff700 1 -- 172.21.15.23:0/444391905 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f74a0077430 con 0x7f74ac0f6820 2021-11-19T08:44:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.815+0000 7f74ab7fe700 1 --2- 172.21.15.23:0/444391905 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f749405c2a0 0x7f749405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.815+0000 7f74ab7fe700 1 --2- 172.21.15.23:0/444391905 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f749405c2a0 0x7f749405e760 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f74ac0f6680 tx=0x7f749c00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:40.818+0000 7f7493fff700 1 -- 172.21.15.23:0/444391905 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74a0049b90 con 0x7f74ac0f6820 2021-11-19T08:44:40.999 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:41.015+0000 7f74b2039700 1 -- 172.21.15.23:0/444391905 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74ac10f250 con 0x7f74ac0f6820 2021-11-19T08:44:41.000 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:41.015+0000 7f7493fff700 1 -- 172.21.15.23:0/444391905 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2953 (secure 0 0 0) 0x7f74a0023070 con 0x7f74ac0f6820 2021-11-19T08:44:41.000 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:41.001 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":84},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:39.410151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:41.002 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:41.017+0000 7f74b2039700 1 -- 172.21.15.23:0/444391905 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f749405c2a0 msgr2=0x7f749405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:41.002 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:41.017+0000 7f74b2039700 1 --2- 172.21.15.23:0/444391905 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f749405c2a0 0x7f749405e760 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f74ac0f6680 tx=0x7f749c00b040).stop 2021-11-19T08:44:41.002 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:41.017+0000 7f74b2039700 1 -- 172.21.15.23:0/444391905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74ac0f6820 msgr2=0x7f74ac109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:41.003 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:41.017+0000 7f74b2039700 1 --2- 172.21.15.23:0/444391905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74ac0f6820 0x7f74ac109520 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f74a0000c00 tx=0x7f74a000b040).stop 2021-11-19T08:44:41.003 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:41.017+0000 7f74b2039700 1 -- 172.21.15.23:0/444391905 shutdown_connections 2021-11-19T08:44:41.003 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:41.017+0000 7f74b2039700 1 --2- 172.21.15.23:0/444391905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74ac0f5580 0x7f74ac109030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:41.003 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:41.017+0000 7f74b2039700 1 --2- 172.21.15.23:0/444391905 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f749405c2a0 0x7f749405e760 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:41.004 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:41.017+0000 7f74b2039700 1 --2- 172.21.15.23:0/444391905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74ac0f6820 0x7f74ac109520 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:41.004 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:41.017+0000 7f74b2039700 1 -- 172.21.15.23:0/444391905 >> 172.21.15.23:0/444391905 conn(0x7f74ac0f0ce0 msgr2=0x7f74ac05fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:41.004 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:41.017+0000 7f74b2039700 1 -- 172.21.15.23:0/444391905 shutdown_connections 2021-11-19T08:44:41.004 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:41.017+0000 7f74b2039700 1 -- 172.21.15.23:0/444391905 wait complete. 2021-11-19T08:44:41.004 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 86 2021-11-19T08:44:41.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:39.410151+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:41.014 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:resolve 2021-11-19T08:44:41.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:41 smithi134 conmon[36363]: audit 2021-11-19T08:44:41.016535+0000 mon.smithi023 (mon.0 2021-11-19T08:44:41.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:41 smithi134 conmon[36363]: ) 863 : audit [DBG] from='client.? 172.21.15.23:0/444391905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:41.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:41 smithi023 conmon[33490]: audit 2021-11-19T08:44:41.016535+0000 mon.smithi023 (mon.0) 863 : audit [DBG] from='client.? 172.21.15.23:0/444391905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:42.015 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:42.150 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.163+0000 7f2f15718700 1 -- 172.21.15.23:0/4131598439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f100f5580 msgr2=0x7f2f100f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:42.150 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.163+0000 7f2f15718700 1 --2- 172.21.15.23:0/4131598439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f100f5580 0x7f2f100f59a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f2f00004660 tx=0x7f2f00009b30).stop 2021-11-19T08:44:42.151 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.163+0000 7f2f15718700 1 -- 172.21.15.23:0/4131598439 shutdown_connections 2021-11-19T08:44:42.151 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.163+0000 7f2f15718700 1 --2- 172.21.15.23:0/4131598439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f100f6820 0x7f2f100f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:42.152 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.163+0000 7f2f15718700 1 --2- 172.21.15.23:0/4131598439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f100f5580 0x7f2f100f59a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:42.152 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.163+0000 7f2f15718700 1 -- 172.21.15.23:0/4131598439 >> 172.21.15.23:0/4131598439 conn(0x7f2f100f0ce0 msgr2=0x7f2f100f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:42.152 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.163+0000 7f2f15718700 1 -- 172.21.15.23:0/4131598439 shutdown_connections 2021-11-19T08:44:42.152 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.163+0000 7f2f15718700 1 -- 172.21.15.23:0/4131598439 wait complete. 2021-11-19T08:44:42.152 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.164+0000 7f2f15718700 1 Processor -- start 2021-11-19T08:44:42.153 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.164+0000 7f2f15718700 1 -- start start 2021-11-19T08:44:42.153 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.164+0000 7f2f15718700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f100f5580 0x7f2f10105db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:42.153 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.164+0000 7f2f15718700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f100f6820 0x7f2f101062a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:42.154 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.164+0000 7f2f15718700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f10106850 con 0x7f2f100f6820 2021-11-19T08:44:42.154 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.164+0000 7f2f15718700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f10106990 con 0x7f2f100f5580 2021-11-19T08:44:42.154 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.164+0000 7f2f0e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f100f6820 0x7f2f101062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:42.155 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.164+0000 7f2f0e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f100f6820 0x7f2f101062a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48592/0 (socket says 172.21.15.23:48592) 2021-11-19T08:44:42.155 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.164+0000 7f2f0e7fc700 1 -- 172.21.15.23:0/1270035114 learned_addr learned my addr 172.21.15.23:0/1270035114 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:42.155 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.164+0000 7f2f0effd700 1 --2- 172.21.15.23:0/1270035114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f100f5580 0x7f2f10105db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:42.156 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.165+0000 7f2f0e7fc700 1 -- 172.21.15.23:0/1270035114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f100f5580 msgr2=0x7f2f10105db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:42.156 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.165+0000 7f2f0e7fc700 1 --2- 172.21.15.23:0/1270035114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f100f5580 0x7f2f10105db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:42.156 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.165+0000 7f2f0e7fc700 1 -- 172.21.15.23:0/1270035114 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f00005040 con 0x7f2f100f6820 2021-11-19T08:44:42.156 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.165+0000 7f2f0e7fc700 1 --2- 172.21.15.23:0/1270035114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f100f6820 0x7f2f101062a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f2f040026d0 tx=0x7f2f04009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:42.156 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.165+0000 7f2ef77fe700 1 -- 172.21.15.23:0/1270035114 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f040079c0 con 0x7f2f100f6820 2021-11-19T08:44:42.157 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.165+0000 7f2f15718700 1 -- 172.21.15.23:0/1270035114 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f1010b430 con 0x7f2f100f6820 2021-11-19T08:44:42.157 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.165+0000 7f2ef77fe700 1 -- 172.21.15.23:0/1270035114 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f0400f040 con 0x7f2f100f6820 2021-11-19T08:44:42.157 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.165+0000 7f2ef77fe700 1 -- 172.21.15.23:0/1270035114 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f040148b0 con 0x7f2f100f6820 2021-11-19T08:44:42.157 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.165+0000 7f2f15718700 1 -- 172.21.15.23:0/1270035114 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f1010b9f0 con 0x7f2f100f6820 2021-11-19T08:44:42.158 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.166+0000 7f2f15718700 1 -- 172.21.15.23:0/1270035114 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f10027a00 con 0x7f2f100f6820 2021-11-19T08:44:42.159 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.166+0000 7f2ef77fe700 1 -- 172.21.15.23:0/1270035114 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2f04007b20 con 0x7f2f100f6820 2021-11-19T08:44:42.159 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.167+0000 7f2ef77fe700 1 --2- 172.21.15.23:0/1270035114 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2ef805c2a0 0x7f2ef805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:42.159 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.167+0000 7f2ef77fe700 1 -- 172.21.15.23:0/1270035114 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f2f040763b0 con 0x7f2f100f6820 2021-11-19T08:44:42.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.167+0000 7f2f0effd700 1 --2- 172.21.15.23:0/1270035114 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2ef805c2a0 0x7f2ef805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:42.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.167+0000 7f2f0effd700 1 --2- 172.21.15.23:0/1270035114 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2ef805c2a0 0x7f2ef805e760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f2f00009f50 tx=0x7f2f00009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:42.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.171+0000 7f2ef77fe700 1 -- 172.21.15.23:0/1270035114 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f04048b70 con 0x7f2f100f6820 2021-11-19T08:44:42.353 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.369+0000 7f2f15718700 1 -- 172.21.15.23:0/1270035114 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f100fdd10 con 0x7f2f100f6820 2021-11-19T08:44:42.354 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.369+0000 7f2ef77fe700 1 -- 172.21.15.23:0/1270035114 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2953 (secure 0 0 0) 0x7f2f0404c190 con 0x7f2f100f6820 2021-11-19T08:44:42.354 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:42.355 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":84},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:39.410151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:42.356 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.371+0000 7f2f15718700 1 -- 172.21.15.23:0/1270035114 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2ef805c2a0 msgr2=0x7f2ef805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:42.356 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.371+0000 7f2f15718700 1 --2- 172.21.15.23:0/1270035114 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2ef805c2a0 0x7f2ef805e760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f2f00009f50 tx=0x7f2f00009b30).stop 2021-11-19T08:44:42.356 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.371+0000 7f2f15718700 1 -- 172.21.15.23:0/1270035114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f100f6820 msgr2=0x7f2f101062a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.371+0000 7f2f15718700 1 --2- 172.21.15.23:0/1270035114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f100f6820 0x7f2f101062a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f2f040026d0 tx=0x7f2f04009de0).stop 2021-11-19T08:44:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.371+0000 7f2f15718700 1 -- 172.21.15.23:0/1270035114 shutdown_connections 2021-11-19T08:44:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.371+0000 7f2f15718700 1 --2- 172.21.15.23:0/1270035114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f100f5580 0x7f2f10105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.371+0000 7f2f15718700 1 --2- 172.21.15.23:0/1270035114 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2ef805c2a0 0x7f2ef805e760 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.371+0000 7f2f15718700 1 --2- 172.21.15.23:0/1270035114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f100f6820 0x7f2f101062a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.371+0000 7f2f15718700 1 -- 172.21.15.23:0/1270035114 >> 172.21.15.23:0/1270035114 conn(0x7f2f100f0ce0 msgr2=0x7f2f100fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.371+0000 7f2f15718700 1 -- 172.21.15.23:0/1270035114 shutdown_connections 2021-11-19T08:44:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:42.371+0000 7f2f15718700 1 -- 172.21.15.23:0/1270035114 wait complete. 2021-11-19T08:44:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 86 2021-11-19T08:44:42.367 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:39.410151+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:42.368 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:resolve 2021-11-19T08:44:42.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:42 smithi023 conmon[33490]: cluster 2021-11-19T08:44:40.893763+0000 mgr.smithi023.ehlpcl (mgr.14164) 320 2021-11-19T08:44:42.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:42 smithi023 conmon[33490]: : cluster [DBG] pgmap v296: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:42.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:42 smithi023 conmon[33490]: audit 2021-11-19T08:44:42.370215+0000 mon.smithi023 (mon.0) 864 : audit [DBG] from='client.? 172.21.15.23:0/1270035114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:42.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:42 smithi134 conmon[36363]: cluster 2021-11-19T08:44:40.893763+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:44:42.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:42 smithi134 conmon[36363]: .14164) 320 : cluster [DBG] pgmap v296: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:42.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:42 smithi134 conmon[36363]: audit 2021-11-19T08:44:42.370215+0000 mon.smithi023 (mon.0) 864 : audit [DBG] from='client.? 172.21.15.23:0/1270035114' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:42.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:42 smithi134 conmon[36363]: 2021-11-19T08:44:43.369 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:43.504 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.516+0000 7fcc57205700 1 -- 172.21.15.23:0/4049947494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc500f7940 msgr2=0x7fcc500f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:43.505 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.516+0000 7fcc57205700 1 --2- 172.21.15.23:0/4049947494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc500f7940 0x7fcc500f7da0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fcc44004660 tx=0x7fcc44009b30).stop 2021-11-19T08:44:43.505 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.516+0000 7fcc57205700 1 -- 172.21.15.23:0/4049947494 shutdown_connections 2021-11-19T08:44:43.506 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.516+0000 7fcc57205700 1 --2- 172.21.15.23:0/4049947494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc500f7940 0x7fcc500f7da0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:43.506 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.516+0000 7fcc57205700 1 --2- 172.21.15.23:0/4049947494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc500f66a0 0x7fcc500f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:43.506 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.516+0000 7fcc57205700 1 -- 172.21.15.23:0/4049947494 >> 172.21.15.23:0/4049947494 conn(0x7fcc500f1da0 msgr2=0x7fcc500f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:43.506 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.516+0000 7fcc57205700 1 -- 172.21.15.23:0/4049947494 shutdown_connections 2021-11-19T08:44:43.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.516+0000 7fcc57205700 1 -- 172.21.15.23:0/4049947494 wait complete. 2021-11-19T08:44:43.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.517+0000 7fcc57205700 1 Processor -- start 2021-11-19T08:44:43.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.517+0000 7fcc57205700 1 -- start start 2021-11-19T08:44:43.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.517+0000 7fcc57205700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc500f66a0 0x7fcc50106e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:43.507 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.517+0000 7fcc57205700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc500f7940 0x7fcc50107340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:43.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.517+0000 7fcc57205700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc501078f0 con 0x7fcc500f66a0 2021-11-19T08:44:43.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.517+0000 7fcc57205700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc50107a30 con 0x7fcc500f7940 2021-11-19T08:44:43.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.517+0000 7fcc54fa1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc500f66a0 0x7fcc50106e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:43.508 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.517+0000 7fcc54fa1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc500f66a0 0x7fcc50106e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48602/0 (socket says 172.21.15.23:48602) 2021-11-19T08:44:43.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.517+0000 7fcc54fa1700 1 -- 172.21.15.23:0/3838842079 learned_addr learned my addr 172.21.15.23:0/3838842079 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:43.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.517+0000 7fcc4ffff700 1 --2- 172.21.15.23:0/3838842079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc500f7940 0x7fcc50107340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:43.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.517+0000 7fcc54fa1700 1 -- 172.21.15.23:0/3838842079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc500f7940 msgr2=0x7fcc50107340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:43.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.517+0000 7fcc54fa1700 1 --2- 172.21.15.23:0/3838842079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc500f7940 0x7fcc50107340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:43.509 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.517+0000 7fcc54fa1700 1 -- 172.21.15.23:0/3838842079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc44005040 con 0x7fcc500f66a0 2021-11-19T08:44:43.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.517+0000 7fcc4ffff700 1 --2- 172.21.15.23:0/3838842079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc500f7940 0x7fcc50107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:44:43.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.518+0000 7fcc54fa1700 1 --2- 172.21.15.23:0/3838842079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc500f66a0 0x7fcc50106e50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fcc4800d7f0 tx=0x7fcc48004e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:43.510 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.518+0000 7fcc4d7fa700 1 -- 172.21.15.23:0/3838842079 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc4800b4f0 con 0x7fcc500f66a0 2021-11-19T08:44:43.511 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.518+0000 7fcc4d7fa700 1 -- 172.21.15.23:0/3838842079 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc4800f040 con 0x7fcc500f66a0 2021-11-19T08:44:43.511 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.518+0000 7fcc57205700 1 -- 172.21.15.23:0/3838842079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5010c4d0 con 0x7fcc500f66a0 2021-11-19T08:44:43.511 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.518+0000 7fcc4d7fa700 1 -- 172.21.15.23:0/3838842079 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc48013980 con 0x7fcc500f66a0 2021-11-19T08:44:43.511 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.518+0000 7fcc57205700 1 -- 172.21.15.23:0/3838842079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5010ca90 con 0x7fcc500f66a0 2021-11-19T08:44:43.511 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.519+0000 7fcc57205700 1 -- 172.21.15.23:0/3838842079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc50027a00 con 0x7fcc500f66a0 2021-11-19T08:44:43.512 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.519+0000 7fcc4d7fa700 1 -- 172.21.15.23:0/3838842079 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcc4801a030 con 0x7fcc500f66a0 2021-11-19T08:44:43.512 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.519+0000 7fcc4d7fa700 1 --2- 172.21.15.23:0/3838842079 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcc3805c2a0 0x7fcc3805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:43.512 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.519+0000 7fcc4d7fa700 1 -- 172.21.15.23:0/3838842079 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7fcc48076260 con 0x7fcc500f66a0 2021-11-19T08:44:43.512 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.519+0000 7fcc4ffff700 1 --2- 172.21.15.23:0/3838842079 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcc3805c2a0 0x7fcc3805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:43.512 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.520+0000 7fcc4ffff700 1 --2- 172.21.15.23:0/3838842079 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcc3805c2a0 0x7fcc3805e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fcc44009d60 tx=0x7fcc44009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:43.513 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.523+0000 7fcc4d7fa700 1 -- 172.21.15.23:0/3838842079 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc48048a20 con 0x7fcc500f66a0 2021-11-19T08:44:43.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:43 smithi023 conmon[33490]: cluster 2021-11-19T08:44:42.430434+0000 mon.smithi023 (mon.0) 865 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3287241569,v1:172.21.15.23:6829/3287241569] up:standby 2021-11-19T08:44:43.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:43 smithi023 conmon[33490]: cluster 2021-11-19T08:44:42.430493+0000 mon.smithi023 (mon.0) 866 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi134.opxllb=up:rejoin} 2 up:standby 2021-11-19T08:44:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.722+0000 7fcc57205700 1 -- 172.21.15.23:0/3838842079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc500fee30 con 0x7fcc500f66a0 2021-11-19T08:44:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.722+0000 7fcc4d7fa700 1 -- 172.21.15.23:0/3838842079 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2952 (secure 0 0 0) 0x7fcc48018090 con 0x7fcc500f66a0 2021-11-19T08:44:43.707 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:43.708 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:39.410151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.724+0000 7fcc57205700 1 -- 172.21.15.23:0/3838842079 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcc3805c2a0 msgr2=0x7fcc3805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:43.709 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.724+0000 7fcc57205700 1 --2- 172.21.15.23:0/3838842079 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcc3805c2a0 0x7fcc3805e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fcc44009d60 tx=0x7fcc44009b30).stop 2021-11-19T08:44:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.724+0000 7fcc57205700 1 -- 172.21.15.23:0/3838842079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc500f66a0 msgr2=0x7fcc50106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.724+0000 7fcc57205700 1 --2- 172.21.15.23:0/3838842079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc500f66a0 0x7fcc50106e50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fcc4800d7f0 tx=0x7fcc48004e80).stop 2021-11-19T08:44:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.724+0000 7fcc57205700 1 -- 172.21.15.23:0/3838842079 shutdown_connections 2021-11-19T08:44:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.724+0000 7fcc57205700 1 --2- 172.21.15.23:0/3838842079 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fcc3805c2a0 0x7fcc3805e760 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:43.711 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.724+0000 7fcc57205700 1 --2- 172.21.15.23:0/3838842079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc500f66a0 0x7fcc50106e50 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:43.711 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.724+0000 7fcc57205700 1 --2- 172.21.15.23:0/3838842079 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc500f7940 0x7fcc50107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:43.711 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.724+0000 7fcc57205700 1 -- 172.21.15.23:0/3838842079 >> 172.21.15.23:0/3838842079 conn(0x7fcc500f1da0 msgr2=0x7fcc500fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:43.711 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.724+0000 7fcc57205700 1 -- 172.21.15.23:0/3838842079 shutdown_connections 2021-11-19T08:44:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:43.724+0000 7fcc57205700 1 -- 172.21.15.23:0/3838842079 wait complete. 2021-11-19T08:44:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 87 2021-11-19T08:44:43.721 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:39.410151+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:43.722 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:resolve 2021-11-19T08:44:43.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:43 smithi134 conmon[36363]: cluster 2021-11-19T08:44:42.430434+0000 mon.smithi023 (mon.0) 865 2021-11-19T08:44:43.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:43 smithi134 conmon[36363]: : cluster [DBG] mds.? [v2:172.21.15.23:6828/3287241569,v1:172.21.15.23:6829/3287241569] up:standby 2021-11-19T08:44:43.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:43 smithi134 conmon[36363]: cluster 2021-11-19T08:44:42.430493+0000 mon.smithi023 (mon.0) 866 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi134.opxllb=up:rejoin} 2 up:standby 2021-11-19T08:44:44.723 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:44.756 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:44 smithi023 conmon[33490]: cluster 2021-11-19T08:44:42 2021-11-19T08:44:44.756 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:44 smithi023 conmon[33490]: .894200+0000 mgr.smithi023.ehlpcl (mgr.14164) 321 : cluster [DBG] pgmap v297: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:44.756 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:44 smithi023 conmon[33490]: audit 2021-11-19T08:44:43.723227+0000 mon.smithi023 (mon.0) 867 : audit [DBG] from='client.? 172.21.15.23:0/3838842079' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:44.756 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:44 smithi023 conmon[33490]: audit 2021-11-19T08:44:43.730878+0000 mon.smithi023 (mon.0) 868 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:44:44.757 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:44 smithi023 conmon[33490]: audit 2021-11-19T08:44:44.117859+0000 mon.smithi023 (mon.0) 869 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:44:44.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:44 smithi134 conmon[36363]: cluster 2021-11-19T08:44:42.894200+0000 mgr.smithi023.ehlpcl (mgr.14164) 321 : cluster [DBG] pgmap v297: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:44.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:44 smithi134 conmon[36363]: audit 2021-11-19T08:44:43.723227+0000 mon.smithi023 (mon.0) 867 : audit [DBG] from='client.? 172.21.15.23:0/3838842079' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:44.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:44 smithi134 conmon[36363]: audit 2021-11-19T08:44:43.730878+0000 mon.smithi023 (mon.0) 868 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-11-19T08:44:44.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:44 smithi134 conmon[36363]: audit 2021-11-19T08:44:44.117859+0000 mon.smithi023 (mon.0) 869 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' 2021-11-19T08:44:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.872+0000 7f755316f700 1 -- 172.21.15.23:0/2061556624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f754c0f6810 msgr2=0x7f754c0f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.872+0000 7f755316f700 1 --2- 172.21.15.23:0/2061556624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f754c0f6810 0x7f754c0f6c70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f753c004660 tx=0x7f753c009b30).stop 2021-11-19T08:44:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.872+0000 7f755316f700 1 -- 172.21.15.23:0/2061556624 shutdown_connections 2021-11-19T08:44:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.872+0000 7f755316f700 1 --2- 172.21.15.23:0/2061556624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f754c0f6810 0x7f754c0f6c70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.872+0000 7f755316f700 1 --2- 172.21.15.23:0/2061556624 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f754c0f5570 0x7f754c0f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.872+0000 7f755316f700 1 -- 172.21.15.23:0/2061556624 >> 172.21.15.23:0/2061556624 conn(0x7f754c0f0cf0 msgr2=0x7f754c0f3110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.872+0000 7f755316f700 1 -- 172.21.15.23:0/2061556624 shutdown_connections 2021-11-19T08:44:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.872+0000 7f755316f700 1 -- 172.21.15.23:0/2061556624 wait complete. 2021-11-19T08:44:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.872+0000 7f755316f700 1 Processor -- start 2021-11-19T08:44:44.860 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.872+0000 7f755316f700 1 -- start start 2021-11-19T08:44:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.873+0000 7f755316f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f754c0f5570 0x7f754c0604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.873+0000 7f755316f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f754c0f6810 0x7f754c10bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.873+0000 7f755316f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c10c100 con 0x7f754c0f6810 2021-11-19T08:44:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.873+0000 7f755316f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c10c240 con 0x7f754c0f5570 2021-11-19T08:44:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.873+0000 7f754bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f754c0f6810 0x7f754c10bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.873+0000 7f754bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f754c0f6810 0x7f754c10bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48614/0 (socket says 172.21.15.23:48614) 2021-11-19T08:44:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.873+0000 7f754bfff700 1 -- 172.21.15.23:0/1742871346 learned_addr learned my addr 172.21.15.23:0/1742871346 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.873+0000 7f7550f0b700 1 --2- 172.21.15.23:0/1742871346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f754c0f5570 0x7f754c0604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.873+0000 7f754bfff700 1 -- 172.21.15.23:0/1742871346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f754c0f5570 msgr2=0x7f754c0604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.873+0000 7f754bfff700 1 --2- 172.21.15.23:0/1742871346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f754c0f5570 0x7f754c0604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.873+0000 7f754bfff700 1 -- 172.21.15.23:0/1742871346 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f753c005040 con 0x7f754c0f6810 2021-11-19T08:44:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.873+0000 7f754bfff700 1 --2- 172.21.15.23:0/1742871346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f754c0f6810 0x7f754c10bc10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f753c000c00 tx=0x7f753c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.874+0000 7f75497fa700 1 -- 172.21.15.23:0/1742871346 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f753c019070 con 0x7f754c0f6810 2021-11-19T08:44:44.863 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.874+0000 7f755316f700 1 -- 172.21.15.23:0/1742871346 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754c10c440 con 0x7f754c0f6810 2021-11-19T08:44:44.864 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.874+0000 7f75497fa700 1 -- 172.21.15.23:0/1742871346 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f753c007e70 con 0x7f754c0f6810 2021-11-19T08:44:44.864 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.874+0000 7f75497fa700 1 -- 172.21.15.23:0/1742871346 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f753c014c90 con 0x7f754c0f6810 2021-11-19T08:44:44.864 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.874+0000 7f755316f700 1 -- 172.21.15.23:0/1742871346 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754c10c970 con 0x7f754c0f6810 2021-11-19T08:44:44.865 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.874+0000 7f755316f700 1 -- 172.21.15.23:0/1742871346 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f754c027a00 con 0x7f754c0f6810 2021-11-19T08:44:44.865 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.875+0000 7f75497fa700 1 -- 172.21.15.23:0/1742871346 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f753c025070 con 0x7f754c0f6810 2021-11-19T08:44:44.866 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.875+0000 7f75497fa700 1 --2- 172.21.15.23:0/1742871346 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f753805c2a0 0x7f753805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:44.866 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.875+0000 7f75497fa700 1 -- 172.21.15.23:0/1742871346 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f753c0774c0 con 0x7f754c0f6810 2021-11-19T08:44:44.866 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.875+0000 7f7550f0b700 1 --2- 172.21.15.23:0/1742871346 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f753805c2a0 0x7f753805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:44.866 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.876+0000 7f7550f0b700 1 --2- 172.21.15.23:0/1742871346 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f753805c2a0 0x7f753805e760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f7544004fd0 tx=0x7f754400b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:44.867 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:44.879+0000 7f75497fa700 1 -- 172.21.15.23:0/1742871346 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f753c049be0 con 0x7f754c0f6810 2021-11-19T08:44:45.058 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:45.074+0000 7f755316f700 1 -- 172.21.15.23:0/1742871346 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f754c0fdd00 con 0x7f754c0f6810 2021-11-19T08:44:45.059 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:45.075+0000 7f75497fa700 1 -- 172.21.15.23:0/1742871346 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f753c04d200 con 0x7f754c0f6810 2021-11-19T08:44:45.060 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:45.060 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:39.410151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:45.061 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:45.076+0000 7f755316f700 1 -- 172.21.15.23:0/1742871346 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f753805c2a0 msgr2=0x7f753805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:45.062 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:45.076+0000 7f755316f700 1 --2- 172.21.15.23:0/1742871346 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f753805c2a0 0x7f753805e760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f7544004fd0 tx=0x7f754400b040).stop 2021-11-19T08:44:45.062 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:45.076+0000 7f755316f700 1 -- 172.21.15.23:0/1742871346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f754c0f6810 msgr2=0x7f754c10bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:45.062 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:45.076+0000 7f755316f700 1 --2- 172.21.15.23:0/1742871346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f754c0f6810 0x7f754c10bc10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f753c000c00 tx=0x7f753c00b040).stop 2021-11-19T08:44:45.062 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:45.076+0000 7f755316f700 1 -- 172.21.15.23:0/1742871346 shutdown_connections 2021-11-19T08:44:45.062 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:45.076+0000 7f755316f700 1 --2- 172.21.15.23:0/1742871346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f754c0f5570 0x7f754c0604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:45.063 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:45.076+0000 7f755316f700 1 --2- 172.21.15.23:0/1742871346 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f753805c2a0 0x7f753805e760 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:45.063 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:45.076+0000 7f755316f700 1 --2- 172.21.15.23:0/1742871346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f754c0f6810 0x7f754c10bc10 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:45.063 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:45.076+0000 7f755316f700 1 -- 172.21.15.23:0/1742871346 >> 172.21.15.23:0/1742871346 conn(0x7f754c0f0cf0 msgr2=0x7f754c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:45.063 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:45.076+0000 7f755316f700 1 -- 172.21.15.23:0/1742871346 shutdown_connections 2021-11-19T08:44:45.064 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:45.076+0000 7f755316f700 1 -- 172.21.15.23:0/1742871346 wait complete. 2021-11-19T08:44:45.064 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 87 2021-11-19T08:44:45.073 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:39.410151+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:45.074 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:resolve 2021-11-19T08:44:45.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:45 smithi134 conmon[36363]: audit 2021-11-19T08:44:45.075661+0000 mon.smithi023 (mon.0) 870 : audit [DBG] 2021-11-19T08:44:45.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:45 smithi134 conmon[36363]: from='client.? 172.21.15.23:0/1742871346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:45.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:45 smithi023 conmon[33490]: audit 2021-11-19T08:44:45.075661+0000 mon.smithi023 ( 2021-11-19T08:44:45.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:45 smithi023 conmon[33490]: mon.0) 870 : audit [DBG] from='client.? 172.21.15.23:0/1742871346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:46.075 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:46.209 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.223+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/1259695786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c680f55a0 msgr2=0x7f4c680f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:46.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.223+0000 7f4c6d7fd700 1 --2- 172.21.15.23:0/1259695786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c680f55a0 0x7f4c680f59c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f4c600099e0 tx=0x7f4c60004d10).stop 2021-11-19T08:44:46.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.223+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/1259695786 shutdown_connections 2021-11-19T08:44:46.210 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.223+0000 7f4c6d7fd700 1 --2- 172.21.15.23:0/1259695786 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c680f6840 0x7f4c680f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:46.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.223+0000 7f4c6d7fd700 1 --2- 172.21.15.23:0/1259695786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c680f55a0 0x7f4c680f59c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:46.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.223+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/1259695786 >> 172.21.15.23:0/1259695786 conn(0x7f4c680f0ce0 msgr2=0x7f4c680f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:46.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.223+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/1259695786 shutdown_connections 2021-11-19T08:44:46.211 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.223+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/1259695786 wait complete. 2021-11-19T08:44:46.212 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c6d7fd700 1 Processor -- start 2021-11-19T08:44:46.212 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c6d7fd700 1 -- start start 2021-11-19T08:44:46.212 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c6d7fd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c680f55a0 0x7f4c6805ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:46.212 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c6d7fd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c680f6840 0x7f4c6810bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:46.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c6d7fd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6810c290 con 0x7f4c680f6840 2021-11-19T08:44:46.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c6d7fd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6810c3d0 con 0x7f4c680f55a0 2021-11-19T08:44:46.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c667fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c680f6840 0x7f4c6810bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:46.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c667fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c680f6840 0x7f4c6810bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48626/0 (socket says 172.21.15.23:48626) 2021-11-19T08:44:46.213 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c667fc700 1 -- 172.21.15.23:0/2767396473 learned_addr learned my addr 172.21.15.23:0/2767396473 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:46.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c66ffd700 1 --2- 172.21.15.23:0/2767396473 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c680f55a0 0x7f4c6805ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:46.214 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c667fc700 1 -- 172.21.15.23:0/2767396473 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c680f55a0 msgr2=0x7f4c6805ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:46.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c667fc700 1 --2- 172.21.15.23:0/2767396473 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c680f55a0 0x7f4c6805ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:46.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c667fc700 1 -- 172.21.15.23:0/2767396473 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c60009640 con 0x7f4c680f6840 2021-11-19T08:44:46.215 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c66ffd700 1 --2- 172.21.15.23:0/2767396473 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c680f55a0 0x7f4c6805ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:44:46.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c667fc700 1 --2- 172.21.15.23:0/2767396473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c680f6840 0x7f4c6810bda0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f4c5000c7c0 tx=0x7f4c50007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:46.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.224+0000 7f4c5f7fe700 1 -- 172.21.15.23:0/2767396473 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c5000e680 con 0x7f4c680f6840 2021-11-19T08:44:46.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.225+0000 7f4c5f7fe700 1 -- 172.21.15.23:0/2767396473 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c5000ec90 con 0x7f4c680f6840 2021-11-19T08:44:46.216 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.225+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/2767396473 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c6810c630 con 0x7f4c680f6840 2021-11-19T08:44:46.217 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.225+0000 7f4c5f7fe700 1 -- 172.21.15.23:0/2767396473 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c50013d50 con 0x7f4c680f6840 2021-11-19T08:44:46.217 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.225+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/2767396473 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6810cb70 con 0x7f4c680f6840 2021-11-19T08:44:46.217 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.225+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/2767396473 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c68027a00 con 0x7f4c680f6840 2021-11-19T08:44:46.219 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.226+0000 7f4c5f7fe700 1 -- 172.21.15.23:0/2767396473 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4c5001a060 con 0x7f4c680f6840 2021-11-19T08:44:46.220 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.226+0000 7f4c5f7fe700 1 --2- 172.21.15.23:0/2767396473 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c5405c2a0 0x7f4c5405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:46.220 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.226+0000 7f4c5f7fe700 1 -- 172.21.15.23:0/2767396473 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f4c50076600 con 0x7f4c680f6840 2021-11-19T08:44:46.220 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.226+0000 7f4c66ffd700 1 --2- 172.21.15.23:0/2767396473 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c5405c2a0 0x7f4c5405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:46.220 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.227+0000 7f4c66ffd700 1 --2- 172.21.15.23:0/2767396473 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c5405c2a0 0x7f4c5405e760 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f4c6000b3b0 tx=0x7f4c60004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:46.221 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.230+0000 7f4c5f7fe700 1 -- 172.21.15.23:0/2767396473 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c50048dc0 con 0x7f4c680f6840 2021-11-19T08:44:46.410 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.425+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/2767396473 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c680fdd30 con 0x7f4c680f6840 2021-11-19T08:44:46.410 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.425+0000 7f4c5f7fe700 1 -- 172.21.15.23:0/2767396473 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f4c5004c3e0 con 0x7f4c680f6840 2021-11-19T08:44:46.411 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:46.411 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:39.410151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:46.413 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.427+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/2767396473 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c5405c2a0 msgr2=0x7f4c5405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:46.413 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.427+0000 7f4c6d7fd700 1 --2- 172.21.15.23:0/2767396473 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c5405c2a0 0x7f4c5405e760 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f4c6000b3b0 tx=0x7f4c60004d10).stop 2021-11-19T08:44:46.413 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.427+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/2767396473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c680f6840 msgr2=0x7f4c6810bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:46.413 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.427+0000 7f4c6d7fd700 1 --2- 172.21.15.23:0/2767396473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c680f6840 0x7f4c6810bda0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f4c5000c7c0 tx=0x7f4c50007d70).stop 2021-11-19T08:44:46.414 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.427+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/2767396473 shutdown_connections 2021-11-19T08:44:46.414 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.427+0000 7f4c6d7fd700 1 --2- 172.21.15.23:0/2767396473 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4c680f55a0 0x7f4c6805ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:46.414 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.427+0000 7f4c6d7fd700 1 --2- 172.21.15.23:0/2767396473 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f4c5405c2a0 0x7f4c5405e760 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:46.414 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.427+0000 7f4c6d7fd700 1 --2- 172.21.15.23:0/2767396473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c680f6840 0x7f4c6810bda0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:46.415 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.427+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/2767396473 >> 172.21.15.23:0/2767396473 conn(0x7f4c680f0ce0 msgr2=0x7f4c680fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:46.416 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.427+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/2767396473 shutdown_connections 2021-11-19T08:44:46.416 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:46.427+0000 7f4c6d7fd700 1 -- 172.21.15.23:0/2767396473 wait complete. 2021-11-19T08:44:46.416 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 87 2021-11-19T08:44:46.424 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:39.410151+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:46.424 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:resolve 2021-11-19T08:44:46.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:46 smithi134 conmon[36363]: cluster 2021-11-19T08:44:44.894612+0000 mgr.smithi023.ehlpcl (mgr.14164) 322 : 2021-11-19T08:44:46.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:46 smithi134 conmon[36363]: cluster [DBG] pgmap v298: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:46.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:46 smithi134 conmon[36363]: audit 2021-11-19T08:44:46.426228+0000 mon.smithi023 (mon.0) 871 : audit [DBG] from='client.? 172.21.15.23:0/2767396473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:46.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:46 smithi023 conmon[33490]: cluster 2021-11-19T08:44:44.894612+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:44:46.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:46 smithi023 conmon[33490]: 14164) 322 : cluster [DBG] pgmap v298: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:46.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:46 smithi023 conmon[33490]: audit 2021-11-19T08:44:46.426228+0000 mon.smithi023 (mon.0) 871 : audit [DBG] from='client.? 172.21.15.23:0/2767396473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:46.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:46 smithi023 conmon[33490]: 2021-11-19T08:44:47.425 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:47.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.574+0000 7f8f93251700 1 -- 172.21.15.23:0/867160083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f8c0f6680 msgr2=0x7f8f8c0f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:47.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.574+0000 7f8f93251700 1 --2- 172.21.15.23:0/867160083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f8c0f6680 0x7f8f8c0f6aa0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f8f80004660 tx=0x7f8f80009b30).stop 2021-11-19T08:44:47.561 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.574+0000 7f8f93251700 1 -- 172.21.15.23:0/867160083 shutdown_connections 2021-11-19T08:44:47.562 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.574+0000 7f8f93251700 1 --2- 172.21.15.23:0/867160083 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8f8c0f7920 0x7f8f8c0f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:47.562 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.574+0000 7f8f93251700 1 --2- 172.21.15.23:0/867160083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f8c0f6680 0x7f8f8c0f6aa0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:47.562 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.574+0000 7f8f93251700 1 -- 172.21.15.23:0/867160083 >> 172.21.15.23:0/867160083 conn(0x7f8f8c0f1de0 msgr2=0x7f8f8c0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:47.562 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.574+0000 7f8f93251700 1 -- 172.21.15.23:0/867160083 shutdown_connections 2021-11-19T08:44:47.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.574+0000 7f8f93251700 1 -- 172.21.15.23:0/867160083 wait complete. 2021-11-19T08:44:47.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.574+0000 7f8f93251700 1 Processor -- start 2021-11-19T08:44:47.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.575+0000 7f8f93251700 1 -- start start 2021-11-19T08:44:47.563 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.575+0000 7f8f93251700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8f8c0f6680 0x7f8f8c106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:47.564 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.575+0000 7f8f93251700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f8c0f7920 0x7f8f8c1073a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:47.564 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.575+0000 7f8f93251700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c107950 con 0x7f8f8c0f7920 2021-11-19T08:44:47.564 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.575+0000 7f8f93251700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c107a90 con 0x7f8f8c0f6680 2021-11-19T08:44:47.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.575+0000 7f8f8bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f8c0f7920 0x7f8f8c1073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:47.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.575+0000 7f8f8bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f8c0f7920 0x7f8f8c1073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48638/0 (socket says 172.21.15.23:48638) 2021-11-19T08:44:47.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.575+0000 7f8f8bfff700 1 -- 172.21.15.23:0/3003634078 learned_addr learned my addr 172.21.15.23:0/3003634078 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:47.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.575+0000 7f8f90fed700 1 --2- 172.21.15.23:0/3003634078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8f8c0f6680 0x7f8f8c106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:47.565 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.575+0000 7f8f8bfff700 1 -- 172.21.15.23:0/3003634078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8f8c0f6680 msgr2=0x7f8f8c106eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:47.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.575+0000 7f8f8bfff700 1 --2- 172.21.15.23:0/3003634078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8f8c0f6680 0x7f8f8c106eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:47.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.575+0000 7f8f8bfff700 1 -- 172.21.15.23:0/3003634078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f80005040 con 0x7f8f8c0f7920 2021-11-19T08:44:47.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.575+0000 7f8f8bfff700 1 --2- 172.21.15.23:0/3003634078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f8c0f7920 0x7f8f8c1073a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f8f780026d0 tx=0x7f8f78009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:47.566 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.576+0000 7f8f897fa700 1 -- 172.21.15.23:0/3003634078 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f780079c0 con 0x7f8f8c0f7920 2021-11-19T08:44:47.567 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.576+0000 7f8f897fa700 1 -- 172.21.15.23:0/3003634078 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f7800f040 con 0x7f8f8c0f7920 2021-11-19T08:44:47.567 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.576+0000 7f8f93251700 1 -- 172.21.15.23:0/3003634078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8c10c530 con 0x7f8f8c0f7920 2021-11-19T08:44:47.567 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.576+0000 7f8f897fa700 1 -- 172.21.15.23:0/3003634078 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f780148b0 con 0x7f8f8c0f7920 2021-11-19T08:44:47.567 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.576+0000 7f8f93251700 1 -- 172.21.15.23:0/3003634078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f8c10caf0 con 0x7f8f8c0f7920 2021-11-19T08:44:47.569 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.577+0000 7f8f93251700 1 -- 172.21.15.23:0/3003634078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f8c027a00 con 0x7f8f8c0f7920 2021-11-19T08:44:47.569 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.577+0000 7f8f897fa700 1 -- 172.21.15.23:0/3003634078 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8f78007b20 con 0x7f8f8c0f7920 2021-11-19T08:44:47.569 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.578+0000 7f8f897fa700 1 --2- 172.21.15.23:0/3003634078 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8f7c05c2a0 0x7f8f7c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:47.569 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.578+0000 7f8f897fa700 1 -- 172.21.15.23:0/3003634078 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f8f78076450 con 0x7f8f8c0f7920 2021-11-19T08:44:47.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.578+0000 7f8f90fed700 1 --2- 172.21.15.23:0/3003634078 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8f7c05c2a0 0x7f8f7c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:47.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.578+0000 7f8f90fed700 1 --2- 172.21.15.23:0/3003634078 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8f7c05c2a0 0x7f8f7c05e760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f8f80005010 tx=0x7f8f80009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:47.570 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.581+0000 7f8f897fa700 1 -- 172.21.15.23:0/3003634078 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f78048b70 con 0x7f8f8c0f7920 2021-11-19T08:44:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.779+0000 7f8f93251700 1 -- 172.21.15.23:0/3003634078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f8c0fee10 con 0x7f8f8c0f7920 2021-11-19T08:44:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.779+0000 7f8f897fa700 1 -- 172.21.15.23:0/3003634078 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f8f7804c190 con 0x7f8f8c0f7920 2021-11-19T08:44:47.768 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:47.768 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:39.410151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:47.769 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.781+0000 7f8f93251700 1 -- 172.21.15.23:0/3003634078 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8f7c05c2a0 msgr2=0x7f8f7c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:47.770 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.781+0000 7f8f93251700 1 --2- 172.21.15.23:0/3003634078 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8f7c05c2a0 0x7f8f7c05e760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f8f80005010 tx=0x7f8f80009b30).stop 2021-11-19T08:44:47.770 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.781+0000 7f8f93251700 1 -- 172.21.15.23:0/3003634078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f8c0f7920 msgr2=0x7f8f8c1073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:47.770 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.781+0000 7f8f93251700 1 --2- 172.21.15.23:0/3003634078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f8c0f7920 0x7f8f8c1073a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f8f780026d0 tx=0x7f8f78009de0).stop 2021-11-19T08:44:47.770 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.781+0000 7f8f93251700 1 -- 172.21.15.23:0/3003634078 shutdown_connections 2021-11-19T08:44:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.781+0000 7f8f93251700 1 --2- 172.21.15.23:0/3003634078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8f8c0f6680 0x7f8f8c106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.781+0000 7f8f93251700 1 --2- 172.21.15.23:0/3003634078 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f8f7c05c2a0 0x7f8f7c05e760 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.781+0000 7f8f93251700 1 --2- 172.21.15.23:0/3003634078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f8c0f7920 0x7f8f8c1073a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.781+0000 7f8f93251700 1 -- 172.21.15.23:0/3003634078 >> 172.21.15.23:0/3003634078 conn(0x7f8f8c0f1de0 msgr2=0x7f8f8c0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:47.772 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.781+0000 7f8f93251700 1 -- 172.21.15.23:0/3003634078 shutdown_connections 2021-11-19T08:44:47.772 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:47.781+0000 7f8f93251700 1 -- 172.21.15.23:0/3003634078 wait complete. 2021-11-19T08:44:47.772 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 87 2021-11-19T08:44:47.777 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:39.410151+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:47.778 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:resolve 2021-11-19T08:44:48.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:48.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:48 smithi134 conmon[36363]: cluster 2021-11-19T08:44:46.895047+0000 mgr.smithi023.ehlpcl (mgr.14164) 323 : cluster [DBG] pgmap v299: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:48.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:48 smithi134 conmon[36363]: 2021-11-19T08:44:48.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:48 smithi134 conmon[36363]: audit 2021-11-19T08:44:47.780423+0000 mon.smithi023 (mon.0) 872 : audit [DBG] from='client.? 172.21.15.23:0/3003634078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:48.807 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:48 smithi023 conmon[33490]: cluster 2021-11-19T08:44:46.895047+0000 mgr.smithi023.ehlpcl (mgr.14164) 323 2021-11-19T08:44:48.808 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:48 smithi023 conmon[33490]: : cluster [DBG] pgmap v299: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:44:48.808 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:48 smithi023 conmon[33490]: audit 2021-11-19T08:44:47.780423+0000 mon.smithi023 (mon.0) 872 2021-11-19T08:44:48.808 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:48 smithi023 conmon[33490]: : audit [DBG] from='client.? 172.21.15.23:0/3003634078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:48.913 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.927+0000 7f7248264700 1 -- 172.21.15.23:0/2640335364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72400f5580 msgr2=0x7f72400f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:48.914 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.927+0000 7f7248264700 1 --2- 172.21.15.23:0/2640335364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72400f5580 0x7f72400f59a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f723c004660 tx=0x7f723c009b30).stop 2021-11-19T08:44:48.914 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.927+0000 7f7248264700 1 -- 172.21.15.23:0/2640335364 shutdown_connections 2021-11-19T08:44:48.914 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.927+0000 7f7248264700 1 --2- 172.21.15.23:0/2640335364 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72400f6820 0x7f72400f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:48.915 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.927+0000 7f7248264700 1 --2- 172.21.15.23:0/2640335364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72400f5580 0x7f72400f59a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:48.915 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.927+0000 7f7248264700 1 -- 172.21.15.23:0/2640335364 >> 172.21.15.23:0/2640335364 conn(0x7f72400f0ce0 msgr2=0x7f72400f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:48.916 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.927+0000 7f7248264700 1 -- 172.21.15.23:0/2640335364 shutdown_connections 2021-11-19T08:44:48.916 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.927+0000 7f7248264700 1 -- 172.21.15.23:0/2640335364 wait complete. 2021-11-19T08:44:48.916 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f7248264700 1 Processor -- start 2021-11-19T08:44:48.916 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f7248264700 1 -- start start 2021-11-19T08:44:48.916 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f7248264700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72400f5580 0x7f72400604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:48.917 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f7248264700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72400f6820 0x7f724010bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:48.917 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f7248264700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724010c190 con 0x7f72400f6820 2021-11-19T08:44:48.917 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f7248264700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724010c2d0 con 0x7f72400f5580 2021-11-19T08:44:48.917 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f72457ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72400f6820 0x7f724010bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:48.918 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f72457ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72400f6820 0x7f724010bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48648/0 (socket says 172.21.15.23:48648) 2021-11-19T08:44:48.918 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f72457ff700 1 -- 172.21.15.23:0/1803211582 learned_addr learned my addr 172.21.15.23:0/1803211582 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:48.918 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f7246000700 1 --2- 172.21.15.23:0/1803211582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72400f5580 0x7f72400604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:48.918 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f72457ff700 1 -- 172.21.15.23:0/1803211582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72400f5580 msgr2=0x7f72400604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:48.918 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f72457ff700 1 --2- 172.21.15.23:0/1803211582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72400f5580 0x7f72400604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:48.919 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f72457ff700 1 -- 172.21.15.23:0/1803211582 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723c005040 con 0x7f72400f6820 2021-11-19T08:44:48.919 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f72457ff700 1 --2- 172.21.15.23:0/1803211582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72400f6820 0x7f724010bca0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f7230002700 tx=0x7f7230009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:48.919 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f72367fc700 1 -- 172.21.15.23:0/1803211582 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7230009320 con 0x7f72400f6820 2021-11-19T08:44:48.919 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.928+0000 7f72367fc700 1 -- 172.21.15.23:0/1803211582 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f723000f040 con 0x7f72400f6820 2021-11-19T08:44:48.920 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.929+0000 7f7248264700 1 -- 172.21.15.23:0/1803211582 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f724010c530 con 0x7f72400f6820 2021-11-19T08:44:48.920 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.929+0000 7f72367fc700 1 -- 172.21.15.23:0/1803211582 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72300147e0 con 0x7f72400f6820 2021-11-19T08:44:48.920 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.929+0000 7f7248264700 1 -- 172.21.15.23:0/1803211582 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f724010caf0 con 0x7f72400f6820 2021-11-19T08:44:48.921 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.930+0000 7f7248264700 1 -- 172.21.15.23:0/1803211582 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7240027a00 con 0x7f72400f6820 2021-11-19T08:44:48.922 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.930+0000 7f72367fc700 1 -- 172.21.15.23:0/1803211582 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7230009480 con 0x7f72400f6820 2021-11-19T08:44:48.922 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.930+0000 7f72367fc700 1 --2- 172.21.15.23:0/1803211582 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f722c05c2a0 0x7f722c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:48.922 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.930+0000 7f72367fc700 1 -- 172.21.15.23:0/1803211582 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f7230076a00 con 0x7f72400f6820 2021-11-19T08:44:48.922 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.931+0000 7f7246000700 1 --2- 172.21.15.23:0/1803211582 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f722c05c2a0 0x7f722c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:48.923 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.931+0000 7f7246000700 1 --2- 172.21.15.23:0/1803211582 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f722c05c2a0 0x7f722c05e760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f723c009f50 tx=0x7f723c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:48.923 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:48.934+0000 7f72367fc700 1 -- 172.21.15.23:0/1803211582 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72300491c0 con 0x7f72400f6820 2021-11-19T08:44:49.117 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:49.132+0000 7f7248264700 1 -- 172.21.15.23:0/1803211582 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72400fdd10 con 0x7f72400f6820 2021-11-19T08:44:49.117 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:49.133+0000 7f72367fc700 1 -- 172.21.15.23:0/1803211582 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f723004c7e0 con 0x7f72400f6820 2021-11-19T08:44:49.118 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:49.118 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:39.410151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:49.120 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:49.134+0000 7f7248264700 1 -- 172.21.15.23:0/1803211582 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f722c05c2a0 msgr2=0x7f722c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:49.120 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:49.134+0000 7f7248264700 1 --2- 172.21.15.23:0/1803211582 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f722c05c2a0 0x7f722c05e760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f723c009f50 tx=0x7f723c009b30).stop 2021-11-19T08:44:49.120 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:49.134+0000 7f7248264700 1 -- 172.21.15.23:0/1803211582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72400f6820 msgr2=0x7f724010bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:49.121 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:49.134+0000 7f7248264700 1 --2- 172.21.15.23:0/1803211582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72400f6820 0x7f724010bca0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f7230002700 tx=0x7f7230009de0).stop 2021-11-19T08:44:49.121 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:49.134+0000 7f7248264700 1 -- 172.21.15.23:0/1803211582 shutdown_connections 2021-11-19T08:44:49.121 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:49.134+0000 7f7248264700 1 --2- 172.21.15.23:0/1803211582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72400f5580 0x7f72400604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:49.121 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:49.134+0000 7f7248264700 1 --2- 172.21.15.23:0/1803211582 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f722c05c2a0 0x7f722c05e760 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:49.122 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:49.134+0000 7f7248264700 1 --2- 172.21.15.23:0/1803211582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72400f6820 0x7f724010bca0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:49.122 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:49.134+0000 7f7248264700 1 -- 172.21.15.23:0/1803211582 >> 172.21.15.23:0/1803211582 conn(0x7f72400f0ce0 msgr2=0x7f72400fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:49.122 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:49.134+0000 7f7248264700 1 -- 172.21.15.23:0/1803211582 shutdown_connections 2021-11-19T08:44:49.123 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:49.135+0000 7f7248264700 1 -- 172.21.15.23:0/1803211582 wait complete. 2021-11-19T08:44:49.123 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 87 2021-11-19T08:44:49.131 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:39.410151+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:49.131 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:resolve 2021-11-19T08:44:49.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:49 smithi134 conmon[36363]: audit 2021-11-19T08:44:49.133835+0000 mon.smithi023 (mon.0) 2021-11-19T08:44:49.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:49 smithi134 conmon[36363]: 873 : audit [DBG] from='client.? 172.21.15.23:0/1803211582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:49.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:49 smithi023 conmon[33490]: audit 2021-11-19T08:44:49.133835+0000 mon.smithi023 (mon 2021-11-19T08:44:49.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:49 smithi023 conmon[33490]: .0) 873 : audit [DBG] from='client.? 172.21.15.23:0/1803211582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:50.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:50.268 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.281+0000 7f304d86b700 1 -- 172.21.15.23:0/2510904807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f304806a350 msgr2=0x7f304806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.281+0000 7f304d86b700 1 --2- 172.21.15.23:0/2510904807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f304806a350 0x7f304806a7d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f3038004660 tx=0x7f3038009b30).stop 2021-11-19T08:44:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.281+0000 7f304d86b700 1 -- 172.21.15.23:0/2510904807 shutdown_connections 2021-11-19T08:44:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.281+0000 7f304d86b700 1 --2- 172.21.15.23:0/2510904807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f304806a350 0x7f304806a7d0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.281+0000 7f304d86b700 1 --2- 172.21.15.23:0/2510904807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f304806b960 0x7f3048069e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:50.270 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.281+0000 7f304d86b700 1 -- 172.21.15.23:0/2510904807 >> 172.21.15.23:0/2510904807 conn(0x7f30480f1de0 msgr2=0x7f30480f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:50.270 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.281+0000 7f304d86b700 1 -- 172.21.15.23:0/2510904807 shutdown_connections 2021-11-19T08:44:50.270 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.281+0000 7f304d86b700 1 -- 172.21.15.23:0/2510904807 wait complete. 2021-11-19T08:44:50.271 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.282+0000 7f304d86b700 1 Processor -- start 2021-11-19T08:44:50.271 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.282+0000 7f304d86b700 1 -- start start 2021-11-19T08:44:50.271 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.282+0000 7f304d86b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f304806a350 0x7f3048108fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:50.271 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.282+0000 7f304d86b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f304806b960 0x7f30481094d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:50.272 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.282+0000 7f304d86b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3048109a80 con 0x7f304806b960 2021-11-19T08:44:50.272 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.282+0000 7f304d86b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3048109bc0 con 0x7f304806a350 2021-11-19T08:44:50.272 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.282+0000 7f30467fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f304806b960 0x7f30481094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:50.272 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.282+0000 7f30467fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f304806b960 0x7f30481094d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48658/0 (socket says 172.21.15.23:48658) 2021-11-19T08:44:50.272 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.282+0000 7f30467fc700 1 -- 172.21.15.23:0/393165781 learned_addr learned my addr 172.21.15.23:0/393165781 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.282+0000 7f3046ffd700 1 --2- 172.21.15.23:0/393165781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f304806a350 0x7f3048108fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.283+0000 7f30467fc700 1 -- 172.21.15.23:0/393165781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f304806a350 msgr2=0x7f3048108fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.283+0000 7f30467fc700 1 --2- 172.21.15.23:0/393165781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f304806a350 0x7f3048108fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:50.273 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.283+0000 7f30467fc700 1 -- 172.21.15.23:0/393165781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3030005040 con 0x7f304806b960 2021-11-19T08:44:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.283+0000 7f30467fc700 1 --2- 172.21.15.23:0/393165781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f304806b960 0x7f30481094d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f3038000c00 tx=0x7f303800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.283+0000 7f303f7fe700 1 -- 172.21.15.23:0/393165781 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3038019070 con 0x7f304806b960 2021-11-19T08:44:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.283+0000 7f304d86b700 1 -- 172.21.15.23:0/393165781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3038005040 con 0x7f304806b960 2021-11-19T08:44:50.274 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.283+0000 7f303f7fe700 1 -- 172.21.15.23:0/393165781 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3038007e70 con 0x7f304806b960 2021-11-19T08:44:50.275 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.283+0000 7f303f7fe700 1 -- 172.21.15.23:0/393165781 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3038014c90 con 0x7f304806b960 2021-11-19T08:44:50.275 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.283+0000 7f304d86b700 1 -- 172.21.15.23:0/393165781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f304810ea30 con 0x7f304806b960 2021-11-19T08:44:50.275 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.284+0000 7f304d86b700 1 -- 172.21.15.23:0/393165781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3048027a00 con 0x7f304806b960 2021-11-19T08:44:50.276 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.284+0000 7f303f7fe700 1 -- 172.21.15.23:0/393165781 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3038025070 con 0x7f304806b960 2021-11-19T08:44:50.276 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.285+0000 7f303f7fe700 1 --2- 172.21.15.23:0/393165781 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f303405c2a0 0x7f303405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:50.277 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.285+0000 7f303f7fe700 1 -- 172.21.15.23:0/393165781 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f3038077420 con 0x7f304806b960 2021-11-19T08:44:50.277 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.285+0000 7f3046ffd700 1 --2- 172.21.15.23:0/393165781 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f303405c2a0 0x7f303405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:50.277 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.285+0000 7f3046ffd700 1 --2- 172.21.15.23:0/393165781 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f303405c2a0 0x7f303405e760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f3030009050 tx=0x7f303000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:50.277 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.289+0000 7f303f7fe700 1 -- 172.21.15.23:0/393165781 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3038049be0 con 0x7f304806b960 2021-11-19T08:44:50.468 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.483+0000 7f304d86b700 1 -- 172.21.15.23:0/393165781 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3048101000 con 0x7f304806b960 2021-11-19T08:44:50.468 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.484+0000 7f303f7fe700 1 -- 172.21.15.23:0/393165781 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f303804d200 con 0x7f304806b960 2021-11-19T08:44:50.469 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:50.469 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:39.410151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:50.470 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.485+0000 7f304d86b700 1 -- 172.21.15.23:0/393165781 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f303405c2a0 msgr2=0x7f303405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:50.471 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.485+0000 7f304d86b700 1 --2- 172.21.15.23:0/393165781 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f303405c2a0 0x7f303405e760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f3030009050 tx=0x7f303000b040).stop 2021-11-19T08:44:50.471 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.485+0000 7f304d86b700 1 -- 172.21.15.23:0/393165781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f304806b960 msgr2=0x7f30481094d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:50.471 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.485+0000 7f304d86b700 1 --2- 172.21.15.23:0/393165781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f304806b960 0x7f30481094d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f3038000c00 tx=0x7f303800b040).stop 2021-11-19T08:44:50.471 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.485+0000 7f304d86b700 1 -- 172.21.15.23:0/393165781 shutdown_connections 2021-11-19T08:44:50.472 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.485+0000 7f304d86b700 1 --2- 172.21.15.23:0/393165781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f304806a350 0x7f3048108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:50.472 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.485+0000 7f304d86b700 1 --2- 172.21.15.23:0/393165781 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f303405c2a0 0x7f303405e760 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:50.472 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.485+0000 7f304d86b700 1 --2- 172.21.15.23:0/393165781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f304806b960 0x7f30481094d0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:50.472 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.485+0000 7f304d86b700 1 -- 172.21.15.23:0/393165781 >> 172.21.15.23:0/393165781 conn(0x7f30480f1de0 msgr2=0x7f30480fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:50.473 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.485+0000 7f304d86b700 1 -- 172.21.15.23:0/393165781 shutdown_connections 2021-11-19T08:44:50.473 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:50.485+0000 7f304d86b700 1 -- 172.21.15.23:0/393165781 wait complete. 2021-11-19T08:44:50.473 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 87 2021-11-19T08:44:50.474 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:50 smithi023 conmon[33490]: cluster 2021-11-19T08:44:48.895434+0000 2021-11-19T08:44:50.474 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:50 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 324 : cluster [DBG] pgmap v300: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-19T08:44:50.474 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:50 smithi023 conmon[33490]: audit 2021-11-19T08:44:50.484714+0000 mon.smithi023 (mon.0) 874 : audit [DBG] from='client.? 172.21.15.23:0/393165781' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:50.474 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:50 smithi023 conmon[33490]: 2021-11-19T08:44:50.482 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:39.410151+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:50.483 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:resolve 2021-11-19T08:44:50.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:50 smithi134 conmon[36363]: cluster 2021-11-19T08:44:48.895434+0000 mgr.smithi023.ehlpcl (mgr.14164) 324 : cluster [DBG] pgmap v300: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-19T08:44:50.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:50 smithi134 conmon[36363]: audit 2021-11-19T08:44:50.484714+0000 mon.smithi023 (mon.0) 874 : audit [DBG] from='client.? 172.21.15.23:0/393165781' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:51.484 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:51.620 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.632+0000 7f7712b9a700 1 -- 172.21.15.23:0/1487272303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f770c0f5570 msgr2=0x7f770c0f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:51.620 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.632+0000 7f7712b9a700 1 --2- 172.21.15.23:0/1487272303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f770c0f5570 0x7f770c0f5990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f7700004660 tx=0x7f7700009b30).stop 2021-11-19T08:44:51.620 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.633+0000 7f7712b9a700 1 -- 172.21.15.23:0/1487272303 shutdown_connections 2021-11-19T08:44:51.620 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.633+0000 7f7712b9a700 1 --2- 172.21.15.23:0/1487272303 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f770c0f6810 0x7f770c0f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:51.621 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.633+0000 7f7712b9a700 1 --2- 172.21.15.23:0/1487272303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f770c0f5570 0x7f770c0f5990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:51.621 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.633+0000 7f7712b9a700 1 -- 172.21.15.23:0/1487272303 >> 172.21.15.23:0/1487272303 conn(0x7f770c0f0cb0 msgr2=0x7f770c0f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:51.621 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.633+0000 7f7712b9a700 1 -- 172.21.15.23:0/1487272303 shutdown_connections 2021-11-19T08:44:51.621 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.633+0000 7f7712b9a700 1 -- 172.21.15.23:0/1487272303 wait complete. 2021-11-19T08:44:51.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.633+0000 7f7712b9a700 1 Processor -- start 2021-11-19T08:44:51.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.633+0000 7f7712b9a700 1 -- start start 2021-11-19T08:44:51.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f7712b9a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f770c0f5570 0x7f770c105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:51.622 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f7712b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f770c0f6810 0x7f770c106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:51.623 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f7712b9a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770c106890 con 0x7f770c0f6810 2021-11-19T08:44:51.623 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f7712b9a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770c1069d0 con 0x7f770c0f5570 2021-11-19T08:44:51.623 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f770bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f770c0f6810 0x7f770c106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:51.623 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f770bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f770c0f6810 0x7f770c106290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48668/0 (socket says 172.21.15.23:48668) 2021-11-19T08:44:51.623 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f770bfff700 1 -- 172.21.15.23:0/2933428846 learned_addr learned my addr 172.21.15.23:0/2933428846 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:51.624 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f7710936700 1 --2- 172.21.15.23:0/2933428846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f770c0f5570 0x7f770c105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:51.624 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f770bfff700 1 -- 172.21.15.23:0/2933428846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f770c0f5570 msgr2=0x7f770c105da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:51.624 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f770bfff700 1 --2- 172.21.15.23:0/2933428846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f770c0f5570 0x7f770c105da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:51.624 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f770bfff700 1 -- 172.21.15.23:0/2933428846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7700005040 con 0x7f770c0f6810 2021-11-19T08:44:51.625 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f770bfff700 1 --2- 172.21.15.23:0/2933428846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f770c0f6810 0x7f770c106290 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f76f80026d0 tx=0x7f76f8009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:51.625 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f77097fa700 1 -- 172.21.15.23:0/2933428846 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76f80079c0 con 0x7f770c0f6810 2021-11-19T08:44:51.625 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f77097fa700 1 -- 172.21.15.23:0/2933428846 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76f800f040 con 0x7f770c0f6810 2021-11-19T08:44:51.625 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f7712b9a700 1 -- 172.21.15.23:0/2933428846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f770c10b3e0 con 0x7f770c0f6810 2021-11-19T08:44:51.626 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.634+0000 7f77097fa700 1 -- 172.21.15.23:0/2933428846 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76f80148b0 con 0x7f770c0f6810 2021-11-19T08:44:51.626 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.635+0000 7f7712b9a700 1 -- 172.21.15.23:0/2933428846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f770c10b9a0 con 0x7f770c0f6810 2021-11-19T08:44:51.626 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.635+0000 7f7712b9a700 1 -- 172.21.15.23:0/2933428846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f770c027a00 con 0x7f770c0f6810 2021-11-19T08:44:51.628 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.636+0000 7f77097fa700 1 -- 172.21.15.23:0/2933428846 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f76f8007b20 con 0x7f770c0f6810 2021-11-19T08:44:51.628 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.636+0000 7f77097fa700 1 --2- 172.21.15.23:0/2933428846 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f76fc05c2a0 0x7f76fc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:51.628 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.636+0000 7f77097fa700 1 -- 172.21.15.23:0/2933428846 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f76f8076340 con 0x7f770c0f6810 2021-11-19T08:44:51.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.636+0000 7f7710936700 1 --2- 172.21.15.23:0/2933428846 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f76fc05c2a0 0x7f76fc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:51.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.637+0000 7f7710936700 1 --2- 172.21.15.23:0/2933428846 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f76fc05c2a0 0x7f76fc05e760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f770c107290 tx=0x7f7700009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:51.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.640+0000 7f77097fa700 1 -- 172.21.15.23:0/2933428846 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76f8048b00 con 0x7f770c0f6810 2021-11-19T08:44:51.821 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.837+0000 7f7712b9a700 1 -- 172.21.15.23:0/2933428846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f770c0fdd00 con 0x7f770c0f6810 2021-11-19T08:44:51.822 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.838+0000 7f77097fa700 1 -- 172.21.15.23:0/2933428846 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f76f804c120 con 0x7f770c0f6810 2021-11-19T08:44:51.823 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:51.823 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:39.410151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:51.824 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.839+0000 7f7712b9a700 1 -- 172.21.15.23:0/2933428846 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f76fc05c2a0 msgr2=0x7f76fc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:51.824 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.839+0000 7f7712b9a700 1 --2- 172.21.15.23:0/2933428846 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f76fc05c2a0 0x7f76fc05e760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f770c107290 tx=0x7f7700009b30).stop 2021-11-19T08:44:51.825 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.839+0000 7f7712b9a700 1 -- 172.21.15.23:0/2933428846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f770c0f6810 msgr2=0x7f770c106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:51.825 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.839+0000 7f7712b9a700 1 --2- 172.21.15.23:0/2933428846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f770c0f6810 0x7f770c106290 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f76f80026d0 tx=0x7f76f8009de0).stop 2021-11-19T08:44:51.825 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.839+0000 7f7712b9a700 1 -- 172.21.15.23:0/2933428846 shutdown_connections 2021-11-19T08:44:51.825 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.839+0000 7f7712b9a700 1 --2- 172.21.15.23:0/2933428846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f770c0f5570 0x7f770c105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:51.826 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.839+0000 7f7712b9a700 1 --2- 172.21.15.23:0/2933428846 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f76fc05c2a0 0x7f76fc05e760 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:51.826 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.839+0000 7f7712b9a700 1 --2- 172.21.15.23:0/2933428846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f770c0f6810 0x7f770c106290 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:51.826 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.839+0000 7f7712b9a700 1 -- 172.21.15.23:0/2933428846 >> 172.21.15.23:0/2933428846 conn(0x7f770c0f0cb0 msgr2=0x7f770c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:51.826 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.839+0000 7f7712b9a700 1 -- 172.21.15.23:0/2933428846 shutdown_connections 2021-11-19T08:44:51.827 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:51.839+0000 7f7712b9a700 1 -- 172.21.15.23:0/2933428846 wait complete. 2021-11-19T08:44:51.827 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 87 2021-11-19T08:44:51.836 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:39.410151+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:51.836 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:resolve 2021-11-19T08:44:52.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:52 smithi134 conmon[36363]: cluster 2021-11-19T08:44:50.896153+0000 mgr.smithi023.ehlpcl (mgr.14164) 325 : cluster [DBG] pgmap v301: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:52.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:52 smithi134 conmon[36363]: audit 2021-11-19T08:44:51.838682+0000 mon.smithi023 (mon.0) 875 : audit [DBG] from='client.? 172.21.15.23:0/2933428846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:52.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:52.866 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:52 smithi023 conmon[33490]: cluster 2021-11-19T08:44:50.896153+0000 2021-11-19T08:44:52.866 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:52 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 325 : cluster [DBG] pgmap v301: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:52.866 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:52 smithi023 conmon[33490]: audit 2021-11-19T08:44:51.838682+0000 mon.smithi023 (mon.0) 875 : audit [DBG] 2021-11-19T08:44:52.867 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:52 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/2933428846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:52.971 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.985+0000 7f2ff9804700 1 -- 172.21.15.23:0/331034177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ff40f62c0 msgr2=0x7f2ff40f6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:52.972 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.985+0000 7f2ff9804700 1 --2- 172.21.15.23:0/331034177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ff40f62c0 0x7f2ff40f6720 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f2fdc004660 tx=0x7f2fdc009b30).stop 2021-11-19T08:44:52.972 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.985+0000 7f2ff9804700 1 -- 172.21.15.23:0/331034177 shutdown_connections 2021-11-19T08:44:52.972 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.985+0000 7f2ff9804700 1 --2- 172.21.15.23:0/331034177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ff40f62c0 0x7f2ff40f6720 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:52.972 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.985+0000 7f2ff9804700 1 --2- 172.21.15.23:0/331034177 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ff40f7cb0 0x7f2ff40f80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:52.973 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.985+0000 7f2ff9804700 1 -- 172.21.15.23:0/331034177 >> 172.21.15.23:0/331034177 conn(0x7f2ff40f1d40 msgr2=0x7f2ff40f4180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:52.973 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.985+0000 7f2ff9804700 1 -- 172.21.15.23:0/331034177 shutdown_connections 2021-11-19T08:44:52.973 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.985+0000 7f2ff9804700 1 -- 172.21.15.23:0/331034177 wait complete. 2021-11-19T08:44:52.973 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.986+0000 7f2ff9804700 1 Processor -- start 2021-11-19T08:44:52.974 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.986+0000 7f2ff9804700 1 -- start start 2021-11-19T08:44:52.974 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.986+0000 7f2ff9804700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ff40f62c0 0x7f2ff4106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:52.974 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.986+0000 7f2ff9804700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ff40f7cb0 0x7f2ff41073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:52.974 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.986+0000 7f2ff9804700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff4107970 con 0x7f2ff40f62c0 2021-11-19T08:44:52.974 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.986+0000 7f2ff9804700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff4108170 con 0x7f2ff40f7cb0 2021-11-19T08:44:52.975 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.986+0000 7f2ff2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ff40f62c0 0x7f2ff4106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:52.975 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.986+0000 7f2ff2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ff40f62c0 0x7f2ff4106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48678/0 (socket says 172.21.15.23:48678) 2021-11-19T08:44:52.975 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.986+0000 7f2ff2ffd700 1 -- 172.21.15.23:0/459663163 learned_addr learned my addr 172.21.15.23:0/459663163 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:52.975 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.986+0000 7f2ff27fc700 1 --2- 172.21.15.23:0/459663163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ff40f7cb0 0x7f2ff41073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:52.976 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.986+0000 7f2ff2ffd700 1 -- 172.21.15.23:0/459663163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ff40f7cb0 msgr2=0x7f2ff41073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:52.976 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.986+0000 7f2ff2ffd700 1 --2- 172.21.15.23:0/459663163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ff40f7cb0 0x7f2ff41073c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:52.976 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.986+0000 7f2ff2ffd700 1 -- 172.21.15.23:0/459663163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc005040 con 0x7f2ff40f62c0 2021-11-19T08:44:52.976 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.986+0000 7f2ff2ffd700 1 --2- 172.21.15.23:0/459663163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ff40f62c0 0x7f2ff4106ed0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f2fec002700 tx=0x7f2fec009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:52.976 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.987+0000 7f2feb7fe700 1 -- 172.21.15.23:0/459663163 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fec0079c0 con 0x7f2ff40f62c0 2021-11-19T08:44:52.977 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.987+0000 7f2feb7fe700 1 -- 172.21.15.23:0/459663163 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fec00f040 con 0x7f2ff40f62c0 2021-11-19T08:44:52.977 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.987+0000 7f2ff9804700 1 -- 172.21.15.23:0/459663163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff4108450 con 0x7f2ff40f62c0 2021-11-19T08:44:52.977 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.987+0000 7f2feb7fe700 1 -- 172.21.15.23:0/459663163 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fec014860 con 0x7f2ff40f62c0 2021-11-19T08:44:52.977 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.987+0000 7f2ff9804700 1 -- 172.21.15.23:0/459663163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff4108a10 con 0x7f2ff40f62c0 2021-11-19T08:44:52.979 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.988+0000 7f2ff9804700 1 -- 172.21.15.23:0/459663163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff40f59f0 con 0x7f2ff40f62c0 2021-11-19T08:44:52.979 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.988+0000 7f2feb7fe700 1 -- 172.21.15.23:0/459663163 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2fec01b060 con 0x7f2ff40f62c0 2021-11-19T08:44:52.979 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.988+0000 7f2feb7fe700 1 --2- 172.21.15.23:0/459663163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2fe005c300 0x7f2fe005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:52.979 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.988+0000 7f2feb7fe700 1 -- 172.21.15.23:0/459663163 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6672+0+0 (secure 0 0 0) 0x7f2fec076210 con 0x7f2ff40f62c0 2021-11-19T08:44:52.980 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.989+0000 7f2ff27fc700 1 --2- 172.21.15.23:0/459663163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2fe005c300 0x7f2fe005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:52.980 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.989+0000 7f2ff27fc700 1 --2- 172.21.15.23:0/459663163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2fe005c300 0x7f2fe005e7c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f2fdc009f50 tx=0x7f2fdc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:52.980 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:52.992+0000 7f2feb7fe700 1 -- 172.21.15.23:0/459663163 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2fec0489d0 con 0x7f2ff40f62c0 2021-11-19T08:44:53.172 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:53.188+0000 7f2ff9804700 1 -- 172.21.15.23:0/459663163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ff4027ab0 con 0x7f2ff40f62c0 2021-11-19T08:44:53.173 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:53.188+0000 7f2feb7fe700 1 -- 172.21.15.23:0/459663163 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2952 (secure 0 0 0) 0x7f2fec04bff0 con 0x7f2ff40f62c0 2021-11-19T08:44:53.173 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:53.174 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":87},{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":86,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:39.410151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24523":{"gid":24523,"name":"cephfs.smithi134.opxllb","rank":1,"incarnation":79,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.134:6827/3725515320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3725515320},{"type":"v1","addr":"172.21.15.134:6827","nonce":3725515320}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:53.175 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:53.189+0000 7f2ff9804700 1 -- 172.21.15.23:0/459663163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2fe005c300 msgr2=0x7f2fe005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:53.175 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:53.190+0000 7f2ff9804700 1 --2- 172.21.15.23:0/459663163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2fe005c300 0x7f2fe005e7c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f2fdc009f50 tx=0x7f2fdc009b30).stop 2021-11-19T08:44:53.175 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:53.190+0000 7f2ff9804700 1 -- 172.21.15.23:0/459663163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ff40f62c0 msgr2=0x7f2ff4106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:53.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:53.190+0000 7f2ff9804700 1 --2- 172.21.15.23:0/459663163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ff40f62c0 0x7f2ff4106ed0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f2fec002700 tx=0x7f2fec009d80).stop 2021-11-19T08:44:53.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:53.190+0000 7f2ff9804700 1 -- 172.21.15.23:0/459663163 shutdown_connections 2021-11-19T08:44:53.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:53.190+0000 7f2ff9804700 1 --2- 172.21.15.23:0/459663163 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f2fe005c300 0x7f2fe005e7c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:53.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:53.190+0000 7f2ff9804700 1 --2- 172.21.15.23:0/459663163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ff40f62c0 0x7f2ff4106ed0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:53.176 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:53.190+0000 7f2ff9804700 1 --2- 172.21.15.23:0/459663163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ff40f7cb0 0x7f2ff41073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:53.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:53.190+0000 7f2ff9804700 1 -- 172.21.15.23:0/459663163 >> 172.21.15.23:0/459663163 conn(0x7f2ff40f1d40 msgr2=0x7f2ff40fb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:53.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:53.190+0000 7f2ff9804700 1 -- 172.21.15.23:0/459663163 shutdown_connections 2021-11-19T08:44:53.177 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:53.190+0000 7f2ff9804700 1 -- 172.21.15.23:0/459663163 wait complete. 2021-11-19T08:44:53.177 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 87 2021-11-19T08:44:53.186 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:39.410151+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24523': {'gid': 24523, 'name': 'cephfs.smithi134.opxllb', 'rank': 1, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.134:6827/3725515320', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 3725515320}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 3725515320}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:53.187 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:resolve 2021-11-19T08:44:53.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:53 smithi134 conmon[36363]: audit 2021-11-19T08:44:53.189139+0000 mon.smithi023 (mon.0) 876 : audit [DBG] from='client.? 172.21.15.23:0/459663163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:53.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:53 smithi023 conmon[33490]: audit 2021-11-19T08:44:53.189139+0000 mon.smithi023 (mon.0) 2021-11-19T08:44:53.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:53 smithi023 conmon[33490]: 876 : audit [DBG] from='client.? 172.21.15.23:0/459663163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:54.189 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:54.332 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.343+0000 7f7688fcb700 1 -- 172.21.15.23:0/4051907199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76840f6820 msgr2=0x7f76840f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:54.332 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.343+0000 7f7688fcb700 1 --2- 172.21.15.23:0/4051907199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76840f6820 0x7f76840f6c80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f7678004660 tx=0x7f7678009b30).stop 2021-11-19T08:44:54.332 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f7688fcb700 1 -- 172.21.15.23:0/4051907199 shutdown_connections 2021-11-19T08:44:54.332 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f7688fcb700 1 --2- 172.21.15.23:0/4051907199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76840f6820 0x7f76840f6c80 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:54.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f7688fcb700 1 --2- 172.21.15.23:0/4051907199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76840f5580 0x7f76840f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:54.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f7688fcb700 1 -- 172.21.15.23:0/4051907199 >> 172.21.15.23:0/4051907199 conn(0x7f76840f0ce0 msgr2=0x7f76840f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:54.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f7688fcb700 1 -- 172.21.15.23:0/4051907199 shutdown_connections 2021-11-19T08:44:54.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f7688fcb700 1 -- 172.21.15.23:0/4051907199 wait complete. 2021-11-19T08:44:54.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f7688fcb700 1 Processor -- start 2021-11-19T08:44:54.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f7688fcb700 1 -- start start 2021-11-19T08:44:54.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f7688fcb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76840f5580 0x7f7684109030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:54.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f7688fcb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76840f6820 0x7f7684109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:54.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f7688fcb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7684109ad0 con 0x7f76840f5580 2021-11-19T08:44:54.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f7688fcb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7684109c10 con 0x7f76840f6820 2021-11-19T08:44:54.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f768259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76840f5580 0x7f7684109030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:54.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f768259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76840f5580 0x7f7684109030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48698/0 (socket says 172.21.15.23:48698) 2021-11-19T08:44:54.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.344+0000 7f768259c700 1 -- 172.21.15.23:0/1202434158 learned_addr learned my addr 172.21.15.23:0/1202434158 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:54.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.345+0000 7f7681d9b700 1 --2- 172.21.15.23:0/1202434158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76840f6820 0x7f7684109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:54.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.345+0000 7f768259c700 1 -- 172.21.15.23:0/1202434158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76840f6820 msgr2=0x7f7684109520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:54.336 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.345+0000 7f768259c700 1 --2- 172.21.15.23:0/1202434158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76840f6820 0x7f7684109520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:54.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.345+0000 7f768259c700 1 -- 172.21.15.23:0/1202434158 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7678005040 con 0x7f76840f5580 2021-11-19T08:44:54.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.345+0000 7f768259c700 1 --2- 172.21.15.23:0/1202434158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76840f5580 0x7f7684109030 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f76740041d0 tx=0x7f7674009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:54.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.345+0000 7f7672ffd700 1 -- 172.21.15.23:0/1202434158 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7674010870 con 0x7f76840f5580 2021-11-19T08:44:54.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.345+0000 7f7672ffd700 1 -- 172.21.15.23:0/1202434158 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7674010e80 con 0x7f76840f5580 2021-11-19T08:44:54.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.345+0000 7f7688fcb700 1 -- 172.21.15.23:0/1202434158 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768410e6b0 con 0x7f76840f5580 2021-11-19T08:44:54.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.345+0000 7f7672ffd700 1 -- 172.21.15.23:0/1202434158 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f767400f3a0 con 0x7f76840f5580 2021-11-19T08:44:54.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.345+0000 7f7688fcb700 1 -- 172.21.15.23:0/1202434158 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768410ec70 con 0x7f76840f5580 2021-11-19T08:44:54.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.346+0000 7f7688fcb700 1 -- 172.21.15.23:0/1202434158 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7684027a00 con 0x7f76840f5580 2021-11-19T08:44:54.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.346+0000 7f7672ffd700 1 -- 172.21.15.23:0/1202434158 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7674021070 con 0x7f76840f5580 2021-11-19T08:44:54.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.347+0000 7f7672ffd700 1 --2- 172.21.15.23:0/1202434158 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f766c05c2a0 0x7f766c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:54.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.347+0000 7f7672ffd700 1 -- 172.21.15.23:0/1202434158 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f7674076f50 con 0x7f76840f5580 2021-11-19T08:44:54.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.347+0000 7f7681d9b700 1 --2- 172.21.15.23:0/1202434158 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f766c05c2a0 0x7f766c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:54.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.347+0000 7f7681d9b700 1 --2- 172.21.15.23:0/1202434158 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f766c05c2a0 0x7f766c05e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f7678004d80 tx=0x7f767800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:54.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.351+0000 7f7672ffd700 1 -- 172.21.15.23:0/1202434158 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76740a32b0 con 0x7f76840f5580 2021-11-19T08:44:54.532 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.548+0000 7f7688fcb700 1 -- 172.21.15.23:0/1202434158 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76840fcc00 con 0x7f76840f5580 2021-11-19T08:44:54.533 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.548+0000 7f7672ffd700 1 -- 172.21.15.23:0/1202434158 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+2573 (secure 0 0 0) 0x7f76740496b0 con 0x7f76840f5580 2021-11-19T08:44:54.534 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:54.534 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78}],"filesystems":[{"mdsmap":{"epoch":88,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:54.234294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:replay","state_seq":2,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.550+0000 7f7688fcb700 1 -- 172.21.15.23:0/1202434158 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f766c05c2a0 msgr2=0x7f766c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:54.536 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.550+0000 7f7688fcb700 1 --2- 172.21.15.23:0/1202434158 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f766c05c2a0 0x7f766c05e760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f7678004d80 tx=0x7f767800b040).stop 2021-11-19T08:44:54.536 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.550+0000 7f7688fcb700 1 -- 172.21.15.23:0/1202434158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76840f5580 msgr2=0x7f7684109030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:54.537 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.550+0000 7f7688fcb700 1 --2- 172.21.15.23:0/1202434158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76840f5580 0x7f7684109030 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f76740041d0 tx=0x7f7674009b30).stop 2021-11-19T08:44:54.537 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.550+0000 7f7688fcb700 1 -- 172.21.15.23:0/1202434158 shutdown_connections 2021-11-19T08:44:54.537 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.550+0000 7f7688fcb700 1 --2- 172.21.15.23:0/1202434158 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f766c05c2a0 0x7f766c05e760 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:54.537 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.550+0000 7f7688fcb700 1 --2- 172.21.15.23:0/1202434158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76840f5580 0x7f7684109030 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:54.538 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.550+0000 7f7688fcb700 1 --2- 172.21.15.23:0/1202434158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76840f6820 0x7f7684109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:54.538 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.550+0000 7f7688fcb700 1 -- 172.21.15.23:0/1202434158 >> 172.21.15.23:0/1202434158 conn(0x7f76840f0ce0 msgr2=0x7f76840fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:54.538 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.550+0000 7f7688fcb700 1 -- 172.21.15.23:0/1202434158 shutdown_connections 2021-11-19T08:44:54.538 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:54.550+0000 7f7688fcb700 1 -- 172.21.15.23:0/1202434158 wait complete. 2021-11-19T08:44:54.538 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 88 2021-11-19T08:44:54.546 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:54.234294+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:replay', 'state_seq': 2, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:54.547 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:resolve 2021-11-19T08:44:54.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:54 smithi134 conmon[36363]: cluster 2021-11-19T08:44:52.896571+0000 mgr.smithi023.ehlpcl (mgr.14164) 326 : cluster [DBG] pgmap v302: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:44:54.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:54 smithi134 conmon[36363]: cluster 2021-11-19T08:44:54.233583+0000 mon.smithi023 (mon.0) 877 : cluster [WRN] Replacing daemon mds.cephfs.smithi134.opxllb as rank 1 with standby daemon mds.cephfs.smithi023.wazanl 2021-11-19T08:44:54.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:54 smithi134 conmon[36363]: cluster 2021-11-19T08:44:54.233636+0000 mon.smithi023 (mon.0) 878 : cluster [INF] MDS daemon mds.cephfs.smithi134.opxllb is removed because it is dead or otherwise unavailable. 2021-11-19T08:44:54.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:54 smithi134 conmon[36363]: cluster 2021-11-19T08:44:54.242114+0000 mon.smithi023 (mon.0) 879 : cluster [DBG] osdmap e57: 6 total, 6 up, 6 in 2021-11-19T08:44:54.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:54 smithi134 conmon[36363]: cluster 2021-11-19T08:44:54.245906 2021-11-19T08:44:54.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:54 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 880 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi023.wazanl=up:replay} 1 up:standby 2021-11-19T08:44:54.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:54 smithi023 conmon[33490]: cluster 2021-11-19T08:44:52.896571+0000 mgr.smithi023.ehlpcl (mgr.14164) 326 : cluster [DBG] pgmap v302: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2021-11-19T08:44:54.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:54 smithi023 conmon[33490]: cluster 2021-11-19T08:44:54.233583+0000 mon.smithi023 (mon.0) 877 : cluster [WRN] Replacing daemon mds.cephfs.smithi134.opxllb as rank 1 with standby daemon mds.cephfs.smithi023.wazanl 2021-11-19T08:44:54.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:54 smithi023 conmon[33490]: cluster 2021-11-19T08:44:54.233636+0000 2021-11-19T08:44:54.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:54 smithi023 conmon[33490]: mon.smithi023 (mon.0) 878 : cluster [INF] MDS daemon mds.cephfs.smithi134.opxllb is removed because it is dead or otherwise unavailable. 2021-11-19T08:44:54.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:54 smithi023 conmon[33490]: cluster 2021-11-19T08:44:54.242114 2021-11-19T08:44:54.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:54 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 879 : cluster [DBG] osdmap e57: 6 total, 6 up, 6 in 2021-11-19T08:44:54.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:54 smithi023 conmon[33490]: cluster 2021-11-19T08:44:54.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:54 smithi023 conmon[33490]: 2021-11-19T08:44:54.245906+0000 mon.smithi023 (mon.0 2021-11-19T08:44:54.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:54 smithi023 conmon[33490]: ) 880 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi023.wazanl=up:replay} 1 up:standby 2021-11-19T08:44:55.548 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:55.685 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.696+0000 7f10f7fff700 1 -- 172.21.15.23:0/3933449099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f80f67a0 msgr2=0x7f10f80f6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:55.686 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.696+0000 7f10f7fff700 1 --2- 172.21.15.23:0/3933449099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f80f67a0 0x7f10f80f6c00 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f10e0004660 tx=0x7f10e0009b30).stop 2021-11-19T08:44:55.686 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.696+0000 7f10f7fff700 1 -- 172.21.15.23:0/3933449099 shutdown_connections 2021-11-19T08:44:55.686 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.696+0000 7f10f7fff700 1 --2- 172.21.15.23:0/3933449099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f80f67a0 0x7f10f80f6c00 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:55.686 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.696+0000 7f10f7fff700 1 --2- 172.21.15.23:0/3933449099 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10f80f5500 0x7f10f80f5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:55.687 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.696+0000 7f10f7fff700 1 -- 172.21.15.23:0/3933449099 >> 172.21.15.23:0/3933449099 conn(0x7f10f80f0c40 msgr2=0x7f10f80f3080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:55.687 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.696+0000 7f10f7fff700 1 -- 172.21.15.23:0/3933449099 shutdown_connections 2021-11-19T08:44:55.687 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.696+0000 7f10f7fff700 1 -- 172.21.15.23:0/3933449099 wait complete. 2021-11-19T08:44:55.687 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.697+0000 7f10f7fff700 1 Processor -- start 2021-11-19T08:44:55.688 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.697+0000 7f10f7fff700 1 -- start start 2021-11-19T08:44:55.688 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.697+0000 7f10f7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f80f5500 0x7f10f8107040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:55.688 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.697+0000 7f10f7fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10f80f67a0 0x7f10f8107530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:55.688 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.697+0000 7f10f7fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f8107ae0 con 0x7f10f80f5500 2021-11-19T08:44:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.697+0000 7f10f7fff700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f8107c20 con 0x7f10f80f67a0 2021-11-19T08:44:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.697+0000 7f10f6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f80f5500 0x7f10f8107040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.697+0000 7f10f6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f80f5500 0x7f10f8107040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48712/0 (socket says 172.21.15.23:48712) 2021-11-19T08:44:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.697+0000 7f10f6ffd700 1 -- 172.21.15.23:0/3646369242 learned_addr learned my addr 172.21.15.23:0/3646369242 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.697+0000 7f10f67fc700 1 --2- 172.21.15.23:0/3646369242 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10f80f67a0 0x7f10f8107530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.698+0000 7f10f6ffd700 1 -- 172.21.15.23:0/3646369242 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10f80f67a0 msgr2=0x7f10f8107530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:55.691 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.698+0000 7f10f6ffd700 1 --2- 172.21.15.23:0/3646369242 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10f80f67a0 0x7f10f8107530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:55.691 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.698+0000 7f10f6ffd700 1 -- 172.21.15.23:0/3646369242 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e0005040 con 0x7f10f80f5500 2021-11-19T08:44:55.691 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.698+0000 7f10f6ffd700 1 --2- 172.21.15.23:0/3646369242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f80f5500 0x7f10f8107040 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f10e800c710 tx=0x7f10e8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:55.691 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.698+0000 7f10ef7fe700 1 -- 172.21.15.23:0/3646369242 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e80106f0 con 0x7f10f80f5500 2021-11-19T08:44:55.692 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.698+0000 7f10ef7fe700 1 -- 172.21.15.23:0/3646369242 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f10e8010d00 con 0x7f10f80f5500 2021-11-19T08:44:55.692 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.698+0000 7f10ef7fe700 1 -- 172.21.15.23:0/3646369242 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e8012d90 con 0x7f10f80f5500 2021-11-19T08:44:55.692 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.698+0000 7f10f7fff700 1 -- 172.21.15.23:0/3646369242 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f81085c0 con 0x7f10f80f5500 2021-11-19T08:44:55.693 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.698+0000 7f10f7fff700 1 -- 172.21.15.23:0/3646369242 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f8108b80 con 0x7f10f80f5500 2021-11-19T08:44:55.693 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.699+0000 7f10f7fff700 1 -- 172.21.15.23:0/3646369242 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f8045ad0 con 0x7f10f80f5500 2021-11-19T08:44:55.693 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.699+0000 7f10ef7fe700 1 -- 172.21.15.23:0/3646369242 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f10e8010070 con 0x7f10f80f5500 2021-11-19T08:44:55.693 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.700+0000 7f10ef7fe700 1 --2- 172.21.15.23:0/3646369242 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f10e405c300 0x7f10e405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:55.693 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.700+0000 7f10ef7fe700 1 -- 172.21.15.23:0/3646369242 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f10e8076930 con 0x7f10f80f5500 2021-11-19T08:44:55.694 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.700+0000 7f10f67fc700 1 --2- 172.21.15.23:0/3646369242 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f10e405c300 0x7f10e405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:55.694 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.700+0000 7f10f67fc700 1 --2- 172.21.15.23:0/3646369242 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f10e405c300 0x7f10e405e7c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f10e0013580 tx=0x7f10e0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.704+0000 7f10ef7fe700 1 -- 172.21.15.23:0/3646369242 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10e8049090 con 0x7f10f80f5500 2021-11-19T08:44:55.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:55 smithi134 conmon[36363]: audit 2021-11-19T08:44:54.549159+0000 mon.smithi023 (mon.0) 881 : audit [DBG] from='client.? 172.21.15.23:0/1202434158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:55.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:55 smithi134 conmon[36363]: cluster 2021-11-19T08:44:55.247631+0000 mon.smithi023 (mon.0) 882 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3287241569,v1:172.21.15.23:6829/3287241569] up:resolve 2021-11-19T08:44:55.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:55 smithi134 conmon[36363]: cluster 2021-11-19T08:44:55.247705+0000 mon.smithi023 (mon.0) 883 : cluster [DBG] mds.? [v2:172.21.15.134:6826/2648154636,v1:172.21.15.134:6827/2648154636] up:boot 2021-11-19T08:44:55.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:55 smithi134 conmon[36363]: cluster 2021-11-19T08:44:55.247782+0000 mon.smithi023 (mon.0) 884 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi023.wazanl=up:resolve} 2 up:standby 2021-11-19T08:44:55.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:55 smithi134 conmon[36363]: audit 2021-11-19T08:44:55.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:55 smithi134 conmon[36363]: 2021-11-19T08:44:55.247926+0000 mon.smithi023 (mon.0) 885 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.opxllb"}]: dispatch 2021-11-19T08:44:55.882 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.898+0000 7f10f7fff700 1 -- 172.21.15.23:0/3646369242 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10f8109220 con 0x7f10f80f5500 2021-11-19T08:44:55.883 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.898+0000 7f10ef7fe700 1 -- 172.21.15.23:0/3646369242 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+2953 (secure 0 0 0) 0x7f10e804c6b0 con 0x7f10f80f5500 2021-11-19T08:44:55.883 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:55.884 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78},{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":89}],"filesystems":[{"mdsmap":{"epoch":89,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:55.243597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:resolve","state_seq":6,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:55.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.900+0000 7f10f7fff700 1 -- 172.21.15.23:0/3646369242 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f10e405c300 msgr2=0x7f10e405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:55.885 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.900+0000 7f10f7fff700 1 --2- 172.21.15.23:0/3646369242 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f10e405c300 0x7f10e405e7c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f10e0013580 tx=0x7f10e0009b30).stop 2021-11-19T08:44:55.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.900+0000 7f10f7fff700 1 -- 172.21.15.23:0/3646369242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f80f5500 msgr2=0x7f10f8107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:55.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.900+0000 7f10f7fff700 1 --2- 172.21.15.23:0/3646369242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f80f5500 0x7f10f8107040 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f10e800c710 tx=0x7f10e8004d10).stop 2021-11-19T08:44:55.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.900+0000 7f10f7fff700 1 -- 172.21.15.23:0/3646369242 shutdown_connections 2021-11-19T08:44:55.886 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.900+0000 7f10f7fff700 1 --2- 172.21.15.23:0/3646369242 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f10e405c300 0x7f10e405e7c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:55.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.900+0000 7f10f7fff700 1 --2- 172.21.15.23:0/3646369242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f80f5500 0x7f10f8107040 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:55.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.900+0000 7f10f7fff700 1 --2- 172.21.15.23:0/3646369242 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10f80f67a0 0x7f10f8107530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:55.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.900+0000 7f10f7fff700 1 -- 172.21.15.23:0/3646369242 >> 172.21.15.23:0/3646369242 conn(0x7f10f80f0c40 msgr2=0x7f10f805ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:55.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.900+0000 7f10f7fff700 1 -- 172.21.15.23:0/3646369242 shutdown_connections 2021-11-19T08:44:55.888 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:55.900+0000 7f10f7fff700 1 -- 172.21.15.23:0/3646369242 wait complete. 2021-11-19T08:44:55.888 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 89 2021-11-19T08:44:55.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:55 smithi023 conmon[33490]: audit 2021-11-19T08:44:54.549159+0000 mon.smithi023 (mon.0) 881 : audit [DBG] from='client.? 172.21.15.23:0/1202434158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:55.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:55 smithi023 conmon[33490]: cluster 2021-11 2021-11-19T08:44:55.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:55 smithi023 conmon[33490]: -19T08:44:55.247631+0000 mon.smithi023 (mon.0) 882 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3287241569,v1:172.21.15.23:6829/3287241569] up:resolve 2021-11-19T08:44:55.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:55 smithi023 conmon[33490]: cluster 2021-11-19T08:44:55.247705+0000 mon.smithi023 (mon.0) 883 : cluster [DBG] mds.? [v2:172.21.15.134:6826/2648154636,v1:172.21.15.134:6827/2648154636] up:boot 2021-11-19T08:44:55.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:55 smithi023 conmon[33490]: cluster 2021-11-19T08:44: 2021-11-19T08:44:55.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:55 smithi023 conmon[33490]: 55.247782+0000 mon.smithi023 (mon.0) 884 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:resolve,1=cephfs.smithi023.wazanl=up:resolve} 2 up:standby 2021-11-19T08:44:55.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:55 smithi023 conmon[33490]: audit 2021-11 2021-11-19T08:44:55.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:55 smithi023 conmon[33490]: -19T08:44:55. 2021-11-19T08:44:55.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:55 smithi023 conmon[33490]: 247926+0000 mon.smithi023 (mon.0) 2021-11-19T08:44:55.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:55 smithi023 conmon[33490]: 885 : audit [DBG] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.opxllb"}]: dispatch 2021-11-19T08:44:55.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:55.243597+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:resolve', 'state_seq': 6, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:55.897 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:resolve 2021-11-19T08:44:56.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:56 smithi134 conmon[36363]: cluster 2021-11-19T08:44:54.897054+0000 mgr.smithi023.ehlpcl (mgr.14164) 327 2021-11-19T08:44:56.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:56 smithi134 conmon[36363]: : cluster [DBG] pgmap v304: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-19T08:44:56.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:56 smithi134 conmon[36363]: audit 2021-11-19T08:44:55.899322+0000 mon.smithi023 (mon.0) 886 : audit [DBG] from='client.? 172.21.15.23:0/3646369242' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:56.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:56 smithi134 conmon[36363]: cluster 2021-11-19T08:44:56.252929+0000 mon.smithi023 (mon.0) 887 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3287241569,v1:172.21.15.23:6829/3287241569] up:reconnect 2021-11-19T08:44:56.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:56 smithi134 conmon[36363]: cluster 2021-11-19T08:44:56.252984+0000 mon.smithi023 (mon.0) 888 : cluster 2021-11-19T08:44:56.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:56 smithi134 conmon[36363]: [DBG] mds.? [v2:172.21.15.23:6826/3968768531,v1:172.21.15.23:6827/3968768531] up:reconnect 2021-11-19T08:44:56.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:56 smithi134 conmon[36363]: cluster 2021-11-19T08 2021-11-19T08:44:56.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:56 smithi134 conmon[36363]: :44:56.253063+0000 mon.smithi023 (mon.0) 889 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:reconnect,1=cephfs.smithi023.wazanl=up:reconnect} 2 up:standby 2021-11-19T08:44:56.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:56 smithi134 conmon[36363]: cluster 2021-11-19T08:44:56.275995+0000 mds.cephfs.smithi023.fuvzkr (mds.0) 2021-11-19T08:44:56.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:56 smithi134 conmon[36363]: 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249998 2021-11-19T08:44:56.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: cluster 2021-11-19T08:44:54.897054+0000 2021-11-19T08:44:56.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: mgr.smithi023.ehlpcl (mgr.14164) 327 : cluster [DBG] pgmap v304: 65 pgs: 65 active+clean; 5.1 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 716 B/s rd, 0 op/s 2021-11-19T08:44:56.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: audit 2021-11-19T08:44:55.899322+0000 mon.smithi023 (mon.0) 2021-11-19T08:44:56.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: 886 : audit [DBG] from='client.? 172.21.15.23:0/3646369242' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:56.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: cluster 2021-11 2021-11-19T08:44:56.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: -19T08:44:56.252929+0000 mon.smithi023 (mon.0) 2021-11-19T08:44:56.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: 887 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3287241569,v1:172.21.15.23:6829/3287241569] up:reconnect 2021-11-19T08:44:56.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:44:56.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: 08:44:56.252984+0000 mon.smithi023 (mon 2021-11-19T08:44:56.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: .0) 888 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3968768531,v1:172.21.15.23:6827/3968768531] up:reconnect 2021-11-19T08:44:56.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: cluster 2021-11 2021-11-19T08:44:56.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: -19T08:44:56.253063+0000 2021-11-19T08:44:56.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: mon.smithi023 (mon.0) 889 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:reconnect,1=cephfs.smithi023.wazanl=up:reconnect} 2 up:standby 2021-11-19T08:44:56.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: cluster 2021-11-19T08:44:56.275995 2021-11-19T08:44:56.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:56 smithi023 conmon[33490]: +0000 mds.cephfs.smithi023.fuvzkr (mds.0) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249998 2021-11-19T08:44:56.898 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:57.037 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.046+0000 7f44fbcfb700 1 -- 172.21.15.23:0/4091176988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f9830 msgr2=0x7f44f40fbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:57.037 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.046+0000 7f44fbcfb700 1 --2- 172.21.15.23:0/4091176988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f9830 0x7f44f40fbc60 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f44e4004660 tx=0x7f44e4009b30).stop 2021-11-19T08:44:57.038 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.046+0000 7f44fbcfb700 1 -- 172.21.15.23:0/4091176988 shutdown_connections 2021-11-19T08:44:57.038 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.046+0000 7f44fbcfb700 1 --2- 172.21.15.23:0/4091176988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f9830 0x7f44f40fbc60 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:57.039 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.046+0000 7f44fbcfb700 1 --2- 172.21.15.23:0/4091176988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44f4060080 0x7f44f40f9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:57.040 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.046+0000 7f44fbcfb700 1 -- 172.21.15.23:0/4091176988 >> 172.21.15.23:0/4091176988 conn(0x7f44f40f0ce0 msgr2=0x7f44f40f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:57.040 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.046+0000 7f44fbcfb700 1 -- 172.21.15.23:0/4091176988 shutdown_connections 2021-11-19T08:44:57.041 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.046+0000 7f44fbcfb700 1 -- 172.21.15.23:0/4091176988 wait complete. 2021-11-19T08:44:57.042 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.046+0000 7f44fbcfb700 1 Processor -- start 2021-11-19T08:44:57.042 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.046+0000 7f44fbcfb700 1 -- start start 2021-11-19T08:44:57.042 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.046+0000 7f44fbcfb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44f4060080 0x7f44f4106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:57.043 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44fbcfb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f9830 0x7f44f41073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:57.043 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44fbcfb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f4107960 con 0x7f44f40f9830 2021-11-19T08:44:57.044 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44fbcfb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f4107aa0 con 0x7f44f4060080 2021-11-19T08:44:57.045 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44f9296700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f9830 0x7f44f41073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:57.045 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44f9296700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f9830 0x7f44f41073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48728/0 (socket says 172.21.15.23:48728) 2021-11-19T08:44:57.046 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44f9296700 1 -- 172.21.15.23:0/1901399919 learned_addr learned my addr 172.21.15.23:0/1901399919 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:57.046 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44f9a97700 1 --2- 172.21.15.23:0/1901399919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44f4060080 0x7f44f4106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:57.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44f9296700 1 -- 172.21.15.23:0/1901399919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44f4060080 msgr2=0x7f44f4106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:57.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44f9296700 1 --2- 172.21.15.23:0/1901399919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44f4060080 0x7f44f4106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:57.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44f9296700 1 -- 172.21.15.23:0/1901399919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e4005040 con 0x7f44f40f9830 2021-11-19T08:44:57.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44f9296700 1 --2- 172.21.15.23:0/1901399919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f9830 0x7f44f41073b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f44e4000c00 tx=0x7f44e4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:57.048 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44ea7fc700 1 -- 172.21.15.23:0/1901399919 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44e4018070 con 0x7f44f40f9830 2021-11-19T08:44:57.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44ea7fc700 1 -- 172.21.15.23:0/1901399919 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44e4013410 con 0x7f44f40f9830 2021-11-19T08:44:57.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44fbcfb700 1 -- 172.21.15.23:0/1901399919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f410c4e0 con 0x7f44f40f9830 2021-11-19T08:44:57.049 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44ea7fc700 1 -- 172.21.15.23:0/1901399919 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44e401c620 con 0x7f44f40f9830 2021-11-19T08:44:57.050 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.047+0000 7f44fbcfb700 1 -- 172.21.15.23:0/1901399919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f410caa0 con 0x7f44f40f9830 2021-11-19T08:44:57.051 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.048+0000 7f44fbcfb700 1 -- 172.21.15.23:0/1901399919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f4027a00 con 0x7f44f40f9830 2021-11-19T08:44:57.051 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.049+0000 7f44ea7fc700 1 -- 172.21.15.23:0/1901399919 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f44e401c8e0 con 0x7f44f40f9830 2021-11-19T08:44:57.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.049+0000 7f44ea7fc700 1 --2- 172.21.15.23:0/1901399919 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f44e005c2a0 0x7f44e005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:57.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.049+0000 7f44ea7fc700 1 -- 172.21.15.23:0/1901399919 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f44e40782c0 con 0x7f44f40f9830 2021-11-19T08:44:57.052 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.049+0000 7f44f9a97700 1 --2- 172.21.15.23:0/1901399919 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f44e005c2a0 0x7f44e005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:57.053 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.050+0000 7f44f9a97700 1 --2- 172.21.15.23:0/1901399919 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f44e005c2a0 0x7f44e005e760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f44f0007bb0 tx=0x7f44f000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:57.054 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.053+0000 7f44ea7fc700 1 -- 172.21.15.23:0/1901399919 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44e404aa20 con 0x7f44f40f9830 2021-11-19T08:44:57.231 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.246+0000 7f44fbcfb700 1 -- 172.21.15.23:0/1901399919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44f40fee40 con 0x7f44f40f9830 2021-11-19T08:44:57.231 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.247+0000 7f44ea7fc700 1 -- 172.21.15.23:0/1901399919 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+2958 (secure 0 0 0) 0x7f44e401de60 con 0x7f44f40f9830 2021-11-19T08:44:57.232 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:57.232 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78},{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":89}],"filesystems":[{"mdsmap":{"epoch":90,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:56.245022+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:57.233 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.248+0000 7f44fbcfb700 1 -- 172.21.15.23:0/1901399919 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f44e005c2a0 msgr2=0x7f44e005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.248+0000 7f44fbcfb700 1 --2- 172.21.15.23:0/1901399919 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f44e005c2a0 0x7f44e005e760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f44f0007bb0 tx=0x7f44f000b040).stop 2021-11-19T08:44:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.248+0000 7f44fbcfb700 1 -- 172.21.15.23:0/1901399919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f9830 msgr2=0x7f44f41073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.248+0000 7f44fbcfb700 1 --2- 172.21.15.23:0/1901399919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f9830 0x7f44f41073b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f44e4000c00 tx=0x7f44e4009b30).stop 2021-11-19T08:44:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.248+0000 7f44fbcfb700 1 -- 172.21.15.23:0/1901399919 shutdown_connections 2021-11-19T08:44:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.248+0000 7f44fbcfb700 1 --2- 172.21.15.23:0/1901399919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44f4060080 0x7f44f4106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.248+0000 7f44fbcfb700 1 --2- 172.21.15.23:0/1901399919 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f44e005c2a0 0x7f44e005e760 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.248+0000 7f44fbcfb700 1 --2- 172.21.15.23:0/1901399919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f9830 0x7f44f41073b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.248+0000 7f44fbcfb700 1 -- 172.21.15.23:0/1901399919 >> 172.21.15.23:0/1901399919 conn(0x7f44f40f0ce0 msgr2=0x7f44f40f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.249+0000 7f44fbcfb700 1 -- 172.21.15.23:0/1901399919 shutdown_connections 2021-11-19T08:44:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:57.249+0000 7f44fbcfb700 1 -- 172.21.15.23:0/1901399919 wait complete. 2021-11-19T08:44:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 90 2021-11-19T08:44:57.247 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:56.245022+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:reconnect', 'state_seq': 11, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:reconnect', 'state_seq': 7, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:57.248 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:reconnect 2021-11-19T08:44:57.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:57 smithi134 conmon[36363]: cluster 2021-11-19T08:44:56.276013+0000 mds.cephfs.smithi023.wazanl (mds.1 2021-11-19T08:44:57.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:57 smithi134 conmon[36363]: ) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249998 2021-11-19T08:44:57.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:57 smithi134 conmon[36363]: audit 2021-11-19T08:44:57.248003+0000 mon.smithi023 (mon.0) 890 : audit [DBG] from='client.? 172.21.15.23:0/1901399919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:57.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:57 smithi134 conmon[36363]: cluster 2021-11-19 2021-11-19T08:44:57.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:57 smithi134 conmon[36363]: T08:44:57.258579+0000 mon.smithi023 (mon.0) 891 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3968768531,v1:172.21.15.23:6827/3968768531] up:rejoin 2021-11-19T08:44:57.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:57 smithi134 conmon[36363]: cluster 2021-11-19T08:44: 2021-11-19T08:44:57.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:57 smithi134 conmon[36363]: 57.258638+0000 mon.smithi023 (mon.0) 892 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3287241569,v1:172.21.15.23:6829/3287241569] up:rejoin 2021-11-19T08:44:57.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:57 smithi134 conmon[36363]: cluster 2021-11-19T08:44:57.258715+0000 mon.smithi023 (mon.0) 2021-11-19T08:44:57.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:57 smithi134 conmon[36363]: 893 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:rejoin,1=cephfs.smithi023.wazanl=up:rejoin} 2 up:standby 2021-11-19T08:44:57.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:57 smithi134 conmon[36363]: cluster 2021-11-19T08:44:57.792 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:57 smithi134 conmon[36363]: 2021-11-19T08:44:57.260336+0000 mon.smithi023 (mon.0) 894 : cluster [INF] daemon mds.cephfs.smithi023.wazanl is now active in filesystem cephfs as rank 1 2021-11-19T08:44:57.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:57 smithi023 conmon[33490]: cluster 2021-11-19T08:44:56.276013+0000 mds.cephfs.smithi023.wazanl (mds.1) 1 : cluster [DBG] reconnect by client.14528 v1:192.168.0.1:0/285381603 after 0.0249998 2021-11-19T08:44:57.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:57 smithi023 conmon[33490]: audit 2021-11-19T08:44:57.248003+0000 mon.smithi023 (mon.0) 890 : audit [DBG] from='client.? 172.21.15.23:0/1901399919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:57.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:57 smithi023 conmon[33490]: cluster 2021-11-19T08:44:57.258579+0000 mon.smithi023 (mon.0) 891 : cluster [DBG] mds.? [v2:172.21.15.23:6826/3968768531,v1:172.21.15.23:6827/3968768531] up:rejoin 2021-11-19T08:44:57.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:57 smithi023 conmon[33490]: cluster 2021-11-19T08:44:57.258638+0000 mon.smithi023 (mon.0 2021-11-19T08:44:57.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:57 smithi023 conmon[33490]: ) 892 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3287241569,v1:172.21.15.23:6829/3287241569] up:rejoin 2021-11-19T08:44:57.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:57 smithi023 conmon[33490]: cluster 2021 2021-11-19T08:44:57.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:57 smithi023 conmon[33490]: -11-19T08:44: 2021-11-19T08:44:57.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:57 smithi023 conmon[33490]: 57.258715+0000 mon.smithi023 (mon.0) 893 2021-11-19T08:44:57.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:57 smithi023 conmon[33490]: : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:rejoin,1=cephfs.smithi023.wazanl=up:rejoin} 2 up:standby 2021-11-19T08:44:57.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:57 smithi023 conmon[33490]: cluster 2021-11-19T08:44:57. 2021-11-19T08:44:57.893 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:57 smithi023 conmon[33490]: 260336+0000 mon.smithi023 (mon.0) 894 : cluster 2021-11-19T08:44:57.894 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:57 smithi023 conmon[33490]: [INF] daemon mds.cephfs.smithi023.wazanl is now active in filesystem cephfs as rank 1 2021-11-19T08:44:58.249 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:58.394 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.407+0000 7fa0739f6700 1 -- 172.21.15.23:0/262963121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa06c0f5580 msgr2=0x7fa06c0f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:58.394 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.407+0000 7fa0739f6700 1 --2- 172.21.15.23:0/262963121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa06c0f5580 0x7fa06c0f59a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7fa05c004660 tx=0x7fa05c009b30).stop 2021-11-19T08:44:58.395 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.407+0000 7fa0739f6700 1 -- 172.21.15.23:0/262963121 shutdown_connections 2021-11-19T08:44:58.395 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.407+0000 7fa0739f6700 1 --2- 172.21.15.23:0/262963121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa06c0f6820 0x7fa06c0f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:58.395 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.407+0000 7fa0739f6700 1 --2- 172.21.15.23:0/262963121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa06c0f5580 0x7fa06c0f59a0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:58.395 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.407+0000 7fa0739f6700 1 -- 172.21.15.23:0/262963121 >> 172.21.15.23:0/262963121 conn(0x7fa06c0f0ce0 msgr2=0x7fa06c0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:58.396 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.407+0000 7fa0739f6700 1 -- 172.21.15.23:0/262963121 shutdown_connections 2021-11-19T08:44:58.396 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.407+0000 7fa0739f6700 1 -- 172.21.15.23:0/262963121 wait complete. 2021-11-19T08:44:58.396 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.408+0000 7fa0739f6700 1 Processor -- start 2021-11-19T08:44:58.396 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.408+0000 7fa0739f6700 1 -- start start 2021-11-19T08:44:58.397 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.408+0000 7fa0739f6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa06c0f5580 0x7fa06c060510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:58.397 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.408+0000 7fa0739f6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa06c0f6820 0x7fa06c10bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:58.397 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.408+0000 7fa0739f6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c10c190 con 0x7fa06c0f6820 2021-11-19T08:44:58.397 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.408+0000 7fa0739f6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c10c2d0 con 0x7fa06c0f5580 2021-11-19T08:44:58.398 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.408+0000 7fa070f91700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa06c0f6820 0x7fa06c10bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:58.398 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.408+0000 7fa070f91700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa06c0f6820 0x7fa06c10bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48740/0 (socket says 172.21.15.23:48740) 2021-11-19T08:44:58.398 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.408+0000 7fa070f91700 1 -- 172.21.15.23:0/2127275620 learned_addr learned my addr 172.21.15.23:0/2127275620 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:58.398 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.408+0000 7fa071792700 1 --2- 172.21.15.23:0/2127275620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa06c0f5580 0x7fa06c060510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:58.398 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.408+0000 7fa070f91700 1 -- 172.21.15.23:0/2127275620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa06c0f5580 msgr2=0x7fa06c060510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:58.399 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.408+0000 7fa070f91700 1 --2- 172.21.15.23:0/2127275620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa06c0f5580 0x7fa06c060510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:58.399 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.408+0000 7fa070f91700 1 -- 172.21.15.23:0/2127275620 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c005040 con 0x7fa06c0f6820 2021-11-19T08:44:58.399 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.409+0000 7fa070f91700 1 --2- 172.21.15.23:0/2127275620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa06c0f6820 0x7fa06c10bca0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7fa068002700 tx=0x7fa068009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:58.399 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.409+0000 7fa061ffb700 1 -- 172.21.15.23:0/2127275620 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa068009320 con 0x7fa06c0f6820 2021-11-19T08:44:58.400 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.409+0000 7fa0739f6700 1 -- 172.21.15.23:0/2127275620 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa06c10c530 con 0x7fa06c0f6820 2021-11-19T08:44:58.400 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.409+0000 7fa061ffb700 1 -- 172.21.15.23:0/2127275620 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa06800f040 con 0x7fa06c0f6820 2021-11-19T08:44:58.400 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.409+0000 7fa061ffb700 1 -- 172.21.15.23:0/2127275620 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0680147e0 con 0x7fa06c0f6820 2021-11-19T08:44:58.400 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.409+0000 7fa0739f6700 1 -- 172.21.15.23:0/2127275620 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa06c10caf0 con 0x7fa06c0f6820 2021-11-19T08:44:58.401 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.410+0000 7fa0739f6700 1 -- 172.21.15.23:0/2127275620 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa06c027a00 con 0x7fa06c0f6820 2021-11-19T08:44:58.402 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.410+0000 7fa061ffb700 1 -- 172.21.15.23:0/2127275620 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa068009480 con 0x7fa06c0f6820 2021-11-19T08:44:58.402 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.411+0000 7fa061ffb700 1 --2- 172.21.15.23:0/2127275620 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa05805c2a0 0x7fa05805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:58.402 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.411+0000 7fa061ffb700 1 -- 172.21.15.23:0/2127275620 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fa068076a60 con 0x7fa06c0f6820 2021-11-19T08:44:58.402 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.411+0000 7fa071792700 1 --2- 172.21.15.23:0/2127275620 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa05805c2a0 0x7fa05805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:58.403 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.411+0000 7fa071792700 1 --2- 172.21.15.23:0/2127275620 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa05805c2a0 0x7fa05805e760 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fa05c009f50 tx=0x7fa05c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:58.403 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.415+0000 7fa061ffb700 1 -- 172.21.15.23:0/2127275620 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0680491c0 con 0x7fa06c0f6820 2021-11-19T08:44:58.599 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.615+0000 7fa0739f6700 1 -- 172.21.15.23:0/2127275620 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa06c0fdd10 con 0x7fa06c0f6820 2021-11-19T08:44:58.600 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.615+0000 7fa061ffb700 1 -- 172.21.15.23:0/2127275620 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+2952 (secure 0 0 0) 0x7fa06804c7e0 con 0x7fa06c0f6820 2021-11-19T08:44:58.600 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:58.601 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78},{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":89}],"filesystems":[{"mdsmap":{"epoch":92,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:58.255999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:58.602 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.617+0000 7fa0739f6700 1 -- 172.21.15.23:0/2127275620 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa05805c2a0 msgr2=0x7fa05805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:58.602 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.617+0000 7fa0739f6700 1 --2- 172.21.15.23:0/2127275620 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa05805c2a0 0x7fa05805e760 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fa05c009f50 tx=0x7fa05c009b30).stop 2021-11-19T08:44:58.603 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.617+0000 7fa0739f6700 1 -- 172.21.15.23:0/2127275620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa06c0f6820 msgr2=0x7fa06c10bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:58.603 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.617+0000 7fa0739f6700 1 --2- 172.21.15.23:0/2127275620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa06c0f6820 0x7fa06c10bca0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7fa068002700 tx=0x7fa068009de0).stop 2021-11-19T08:44:58.603 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.617+0000 7fa0739f6700 1 -- 172.21.15.23:0/2127275620 shutdown_connections 2021-11-19T08:44:58.603 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.617+0000 7fa0739f6700 1 --2- 172.21.15.23:0/2127275620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa06c0f5580 0x7fa06c060510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:58.604 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.617+0000 7fa0739f6700 1 --2- 172.21.15.23:0/2127275620 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa05805c2a0 0x7fa05805e760 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:58.604 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.617+0000 7fa0739f6700 1 --2- 172.21.15.23:0/2127275620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa06c0f6820 0x7fa06c10bca0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:58.604 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.617+0000 7fa0739f6700 1 -- 172.21.15.23:0/2127275620 >> 172.21.15.23:0/2127275620 conn(0x7fa06c0f0ce0 msgr2=0x7fa06c0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:58.604 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.617+0000 7fa0739f6700 1 -- 172.21.15.23:0/2127275620 shutdown_connections 2021-11-19T08:44:58.604 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:58.617+0000 7fa0739f6700 1 -- 172.21.15.23:0/2127275620 wait complete. 2021-11-19T08:44:58.605 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 92 2021-11-19T08:44:58.614 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:58.255999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:58.615 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:rejoin 2021-11-19T08:44:58.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:58 smithi134 conmon[36363]: cluster 2021-11-19T08:44:56.897588+0000 mgr.smithi023.ehlpcl (mgr.14164) 328 : cluster [DBG] pgmap v305: 65 pgs: 65 active+clean; 5.1 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:58.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:58 smithi134 conmon[36363]: cluster 2021-11-19T08:44:58.264621+0000 mon.smithi023 (mon.0) 895 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3287241569,v1:172.21.15.23:6829/3287241569] up:active 2021-11-19T08:44:58.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:58 smithi134 conmon[36363]: cluster 2021-11-19T08:44:58.264723+0000 mon.smithi023 (mon.0) 896 : cluster 2021-11-19T08:44:58.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:58 smithi134 conmon[36363]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:rejoin,1=cephfs.smithi023.wazanl=up:active} 2 up:standby 2021-11-19T08:44:58.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:58 smithi023 conmon[33490]: cluster 2021-11-19T08:44:56.897588+0000 mgr.smithi023.ehlpcl (mgr.14164) 328 : cluster [DBG] pgmap v305: 65 pgs: 65 active+clean; 5.1 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:44:58.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:58 smithi023 conmon[33490]: cluster 2021-11-19T08:44:58.264621+0000 mon.smithi023 (mon.0) 895 : cluster [DBG] mds.? [v2:172.21.15.23:6828/3287241569,v1:172.21.15.23:6829/3287241569] up:active 2021-11-19T08:44:58.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:58 smithi023 conmon[33490]: cluster 2021-11-19T08:44:58.264723+0000 mon.smithi023 (mon.0) 896 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:rejoin,1=cephfs.smithi023.wazanl=up:active} 2 up:standby 2021-11-19T08:44:59.616 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:44:59.751 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.764+0000 7fa27e0c6700 1 -- 172.21.15.23:0/725551291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2780f7920 msgr2=0x7fa2780f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.764+0000 7fa27e0c6700 1 --2- 172.21.15.23:0/725551291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2780f7920 0x7fa2780f7d80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7fa26c004660 tx=0x7fa26c009b30).stop 2021-11-19T08:44:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.764+0000 7fa27e0c6700 1 -- 172.21.15.23:0/725551291 shutdown_connections 2021-11-19T08:44:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.764+0000 7fa27e0c6700 1 --2- 172.21.15.23:0/725551291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2780f7920 0x7fa2780f7d80 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.764+0000 7fa27e0c6700 1 --2- 172.21.15.23:0/725551291 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2780f6680 0x7fa2780f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:59.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.764+0000 7fa27e0c6700 1 -- 172.21.15.23:0/725551291 >> 172.21.15.23:0/725551291 conn(0x7fa2780f1de0 msgr2=0x7fa2780f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:59.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.764+0000 7fa27e0c6700 1 -- 172.21.15.23:0/725551291 shutdown_connections 2021-11-19T08:44:59.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.764+0000 7fa27e0c6700 1 -- 172.21.15.23:0/725551291 wait complete. 2021-11-19T08:44:59.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.765+0000 7fa27e0c6700 1 Processor -- start 2021-11-19T08:44:59.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.765+0000 7fa27e0c6700 1 -- start start 2021-11-19T08:44:59.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.765+0000 7fa27e0c6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2780f6680 0x7fa278106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:59.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.765+0000 7fa27e0c6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2780f7920 0x7fa278107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:59.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.765+0000 7fa27e0c6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa278107930 con 0x7fa2780f6680 2021-11-19T08:44:59.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.765+0000 7fa27e0c6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa278107a70 con 0x7fa2780f7920 2021-11-19T08:44:59.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.765+0000 7fa2777fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2780f6680 0x7fa278106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:59.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.765+0000 7fa2777fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2780f6680 0x7fa278106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48758/0 (socket says 172.21.15.23:48758) 2021-11-19T08:44:59.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.765+0000 7fa2777fe700 1 -- 172.21.15.23:0/3732887825 learned_addr learned my addr 172.21.15.23:0/3732887825 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:44:59.756 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.766+0000 7fa276ffd700 1 --2- 172.21.15.23:0/3732887825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2780f7920 0x7fa278107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:59.756 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.766+0000 7fa2777fe700 1 -- 172.21.15.23:0/3732887825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2780f7920 msgr2=0x7fa278107380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:59.756 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.766+0000 7fa2777fe700 1 --2- 172.21.15.23:0/3732887825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2780f7920 0x7fa278107380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:59.757 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.766+0000 7fa2777fe700 1 -- 172.21.15.23:0/3732887825 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c005040 con 0x7fa2780f6680 2021-11-19T08:44:59.757 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.766+0000 7fa276ffd700 1 --2- 172.21.15.23:0/3732887825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2780f7920 0x7fa278107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:44:59.757 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.766+0000 7fa2777fe700 1 --2- 172.21.15.23:0/3732887825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2780f6680 0x7fa278106e90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fa268002700 tx=0x7fa268009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:59.757 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.766+0000 7fa25ffff700 1 -- 172.21.15.23:0/3732887825 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa268009320 con 0x7fa2780f6680 2021-11-19T08:44:59.757 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.766+0000 7fa25ffff700 1 -- 172.21.15.23:0/3732887825 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa26800f040 con 0x7fa2780f6680 2021-11-19T08:44:59.758 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.766+0000 7fa27e0c6700 1 -- 172.21.15.23:0/3732887825 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27810c510 con 0x7fa2780f6680 2021-11-19T08:44:59.758 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.766+0000 7fa25ffff700 1 -- 172.21.15.23:0/3732887825 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa268014900 con 0x7fa2780f6680 2021-11-19T08:44:59.758 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.766+0000 7fa27e0c6700 1 -- 172.21.15.23:0/3732887825 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27810cad0 con 0x7fa2780f6680 2021-11-19T08:44:59.758 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.767+0000 7fa27e0c6700 1 -- 172.21.15.23:0/3732887825 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa278027a00 con 0x7fa2780f6680 2021-11-19T08:44:59.760 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.767+0000 7fa25ffff700 1 -- 172.21.15.23:0/3732887825 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa268009480 con 0x7fa2780f6680 2021-11-19T08:44:59.760 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.768+0000 7fa25ffff700 1 --2- 172.21.15.23:0/3732887825 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa26005c2a0 0x7fa26005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:44:59.760 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.768+0000 7fa25ffff700 1 -- 172.21.15.23:0/3732887825 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fa268076780 con 0x7fa2780f6680 2021-11-19T08:44:59.760 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.768+0000 7fa276ffd700 1 --2- 172.21.15.23:0/3732887825 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa26005c2a0 0x7fa26005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:44:59.761 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.768+0000 7fa276ffd700 1 --2- 172.21.15.23:0/3732887825 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa26005c2a0 0x7fa26005e760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fa26c005010 tx=0x7fa26c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:44:59.761 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.772+0000 7fa25ffff700 1 -- 172.21.15.23:0/3732887825 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa268048ee0 con 0x7fa2780f6680 2021-11-19T08:44:59.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:59 smithi134 conmon[36363]: audit 2021-11-19T08:44:58.616165+0000 mon.smithi023 (mon.0) 897 : audit [DBG] from='client.? 172.21.15.23:0/2127275620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:59.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:59 smithi134 conmon[36363]: 2021-11-19T08:44:59.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:59 smithi134 conmon[36363]: cluster 2021-11-19T08:44:59.265467+0000 mon.smithi023 (mon.0) 898 : cluster [DBG] mds.? [v2:172.21.15.134:6826/2648154636,v1:172.21.15.134:6827/2648154636] up:standby 2021-11-19T08:44:59.791 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:44:59 smithi134 conmon[36363]: cluster 2021-11-19T08:44:59.265551+0000 mon.smithi023 (mon.0) 899 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:rejoin,1=cephfs.smithi023.wazanl=up:active} 2 up:standby 2021-11-19T08:44:59.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:59 smithi023 conmon[33490]: audit 2021-11-19T08:44:58.616165+0000 mon.smithi023 (mon.0) 897 : audit [DBG] from='client.? 172.21.15.23:0/2127275620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:44:59.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:59 smithi023 conmon[33490]: cluster 2021-11-19T08:44:59.265467+0000 mon.smithi023 (mon.0) 898 : cluster [DBG] mds.? [v2:172.21.15.134:6826/2648154636,v1:172.21.15.134:6827/2648154636] up:standby 2021-11-19T08:44:59.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:59 smithi023 conmon[33490]: cluster 2021-11-19T08:44:59.265551+0000 mon.smithi023 (mon.0) 899 : cluster 2021-11-19T08:44:59.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:44:59 smithi023 conmon[33490]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi023.fuvzkr=up:rejoin,1=cephfs.smithi023.wazanl=up:active} 2 up:standby 2021-11-19T08:44:59.951 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.967+0000 7fa27e0c6700 1 -- 172.21.15.23:0/3732887825 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2780fee10 con 0x7fa2780f6680 2021-11-19T08:44:59.952 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.967+0000 7fa25ffff700 1 -- 172.21.15.23:0/3732887825 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2951 (secure 0 0 0) 0x7fa26804c500 con 0x7fa2780f6680 2021-11-19T08:44:59.952 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:44:59.952 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78},{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:58.255999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:44:59.954 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.969+0000 7fa27e0c6700 1 -- 172.21.15.23:0/3732887825 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa26005c2a0 msgr2=0x7fa26005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:59.954 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.969+0000 7fa27e0c6700 1 --2- 172.21.15.23:0/3732887825 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa26005c2a0 0x7fa26005e760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fa26c005010 tx=0x7fa26c009b30).stop 2021-11-19T08:44:59.954 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.969+0000 7fa27e0c6700 1 -- 172.21.15.23:0/3732887825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2780f6680 msgr2=0x7fa278106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:44:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.969+0000 7fa27e0c6700 1 --2- 172.21.15.23:0/3732887825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2780f6680 0x7fa278106e90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7fa268002700 tx=0x7fa268009d80).stop 2021-11-19T08:44:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.969+0000 7fa27e0c6700 1 -- 172.21.15.23:0/3732887825 shutdown_connections 2021-11-19T08:44:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.969+0000 7fa27e0c6700 1 --2- 172.21.15.23:0/3732887825 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa26005c2a0 0x7fa26005e760 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.969+0000 7fa27e0c6700 1 --2- 172.21.15.23:0/3732887825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2780f6680 0x7fa278106e90 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:59.956 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.969+0000 7fa27e0c6700 1 --2- 172.21.15.23:0/3732887825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2780f7920 0x7fa278107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:44:59.956 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.969+0000 7fa27e0c6700 1 -- 172.21.15.23:0/3732887825 >> 172.21.15.23:0/3732887825 conn(0x7fa2780f1de0 msgr2=0x7fa2780fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:44:59.956 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.969+0000 7fa27e0c6700 1 -- 172.21.15.23:0/3732887825 shutdown_connections 2021-11-19T08:44:59.956 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:44:59.969+0000 7fa27e0c6700 1 -- 172.21.15.23:0/3732887825 wait complete. 2021-11-19T08:44:59.956 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 93 2021-11-19T08:44:59.966 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:58.255999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:44:59.966 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:rejoin 2021-11-19T08:45:00.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:00 smithi134 conmon[36363]: cluster 2021-11-19T08:44:58.898207+0000 mgr.smithi023.ehlpcl (mgr.14164) 329 : 2021-11-19T08:45:00.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:00 smithi134 conmon[36363]: cluster [DBG] pgmap v306: 65 pgs: 65 active+clean; 5.1 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:45:00.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:00 smithi134 conmon[36363]: audit 2021-11-19T08:44:59.968151+0000 mon.smithi023 (mon.0) 900 : audit [DBG] from='client.? 172.21.15.23:0/3732887825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:00.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:00 smithi023 conmon[33490]: cluster 2021-11-19T08:44:58.898207+0000 mgr.smithi023.ehlpcl (mgr.14164 2021-11-19T08:45:00.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:00 smithi023 conmon[33490]: ) 329 : cluster [DBG] pgmap v306: 65 pgs: 65 active+clean; 5.1 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-11-19T08:45:00.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:00 smithi023 conmon[33490]: audit 2021-11-19T08:44:59.968151+0000 mon.smithi023 (mon.0) 900 : audit 2021-11-19T08:45:00.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:00 smithi023 conmon[33490]: [DBG] from='client.? 172.21.15.23:0/3732887825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:00.968 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:01.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.116+0000 7f19b5343700 1 -- 172.21.15.23:0/2958764895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19b00f66a0 msgr2=0x7f19b00f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:01.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.116+0000 7f19b5343700 1 --2- 172.21.15.23:0/2958764895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19b00f66a0 0x7f19b00f6ac0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f19a0004660 tx=0x7f19a0009b30).stop 2021-11-19T08:45:01.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.116+0000 7f19b5343700 1 -- 172.21.15.23:0/2958764895 shutdown_connections 2021-11-19T08:45:01.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.116+0000 7f19b5343700 1 --2- 172.21.15.23:0/2958764895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19b00f7940 0x7f19b00f7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:01.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.116+0000 7f19b5343700 1 --2- 172.21.15.23:0/2958764895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19b00f66a0 0x7f19b00f6ac0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:01.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.116+0000 7f19b5343700 1 -- 172.21.15.23:0/2958764895 >> 172.21.15.23:0/2958764895 conn(0x7f19b00f1da0 msgr2=0x7f19b00f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:01.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.116+0000 7f19b5343700 1 -- 172.21.15.23:0/2958764895 shutdown_connections 2021-11-19T08:45:01.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.116+0000 7f19b5343700 1 -- 172.21.15.23:0/2958764895 wait complete. 2021-11-19T08:45:01.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.117+0000 7f19b5343700 1 Processor -- start 2021-11-19T08:45:01.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.117+0000 7f19b5343700 1 -- start start 2021-11-19T08:45:01.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.117+0000 7f19b5343700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19b00f66a0 0x7f19b006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:01.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.117+0000 7f19b5343700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19b00f7940 0x7f19b0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:01.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.117+0000 7f19b5343700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b006bd70 con 0x7f19b00f66a0 2021-11-19T08:45:01.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.117+0000 7f19b5343700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b006a310 con 0x7f19b00f7940 2021-11-19T08:45:01.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.117+0000 7f19aeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19b00f66a0 0x7f19b006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:01.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.117+0000 7f19aeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19b00f66a0 0x7f19b006b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48770/0 (socket says 172.21.15.23:48770) 2021-11-19T08:45:01.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.117+0000 7f19aeffd700 1 -- 172.21.15.23:0/1114571273 learned_addr learned my addr 172.21.15.23:0/1114571273 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:01.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.117+0000 7f19ae7fc700 1 --2- 172.21.15.23:0/1114571273 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19b00f7940 0x7f19b0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.118+0000 7f19aeffd700 1 -- 172.21.15.23:0/1114571273 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19b00f7940 msgr2=0x7f19b0069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.118+0000 7f19aeffd700 1 --2- 172.21.15.23:0/1114571273 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19b00f7940 0x7f19b0069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.118+0000 7f19aeffd700 1 -- 172.21.15.23:0/1114571273 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a0005040 con 0x7f19b00f66a0 2021-11-19T08:45:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.118+0000 7f19ae7fc700 1 --2- 172.21.15.23:0/1114571273 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19b00f7940 0x7f19b0069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:45:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.118+0000 7f19aeffd700 1 --2- 172.21.15.23:0/1114571273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19b00f66a0 0x7f19b006b7c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f19a0004e30 tx=0x7f19a0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:01.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.118+0000 7f19977fe700 1 -- 172.21.15.23:0/1114571273 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19a0018070 con 0x7f19b00f66a0 2021-11-19T08:45:01.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.118+0000 7f19977fe700 1 -- 172.21.15.23:0/1114571273 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19a0013410 con 0x7f19b00f66a0 2021-11-19T08:45:01.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.118+0000 7f19977fe700 1 -- 172.21.15.23:0/1114571273 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19a001c3c0 con 0x7f19b00f66a0 2021-11-19T08:45:01.109 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.118+0000 7f19b5343700 1 -- 172.21.15.23:0/1114571273 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b006a530 con 0x7f19b00f66a0 2021-11-19T08:45:01.110 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.118+0000 7f19b5343700 1 -- 172.21.15.23:0/1114571273 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b006aa90 con 0x7f19b00f66a0 2021-11-19T08:45:01.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.119+0000 7f19b5343700 1 -- 172.21.15.23:0/1114571273 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b0046330 con 0x7f19b00f66a0 2021-11-19T08:45:01.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.119+0000 7f19977fe700 1 -- 172.21.15.23:0/1114571273 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f19a001d420 con 0x7f19b00f66a0 2021-11-19T08:45:01.111 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.120+0000 7f19977fe700 1 --2- 172.21.15.23:0/1114571273 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f199805c2a0 0x7f199805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:01.112 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.120+0000 7f19977fe700 1 -- 172.21.15.23:0/1114571273 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f19a00780c0 con 0x7f19b00f66a0 2021-11-19T08:45:01.112 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.120+0000 7f19ae7fc700 1 --2- 172.21.15.23:0/1114571273 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f199805c2a0 0x7f199805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:01.112 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.120+0000 7f19ae7fc700 1 --2- 172.21.15.23:0/1114571273 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f199805c2a0 0x7f199805e760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f19a4009740 tx=0x7f19a4009be0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:01.112 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.124+0000 7f19977fe700 1 -- 172.21.15.23:0/1114571273 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f19a004a820 con 0x7f19b00f66a0 2021-11-19T08:45:01.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.324+0000 7f19b5343700 1 -- 172.21.15.23:0/1114571273 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19b00fee30 con 0x7f19b00f66a0 2021-11-19T08:45:01.309 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.325+0000 7f19977fe700 1 -- 172.21.15.23:0/1114571273 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f19a004de40 con 0x7f19b00f66a0 2021-11-19T08:45:01.310 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:01.310 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78},{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:58.255999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:01.312 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.326+0000 7f19b5343700 1 -- 172.21.15.23:0/1114571273 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f199805c2a0 msgr2=0x7f199805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:01.312 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.326+0000 7f19b5343700 1 --2- 172.21.15.23:0/1114571273 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f199805c2a0 0x7f199805e760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f19a4009740 tx=0x7f19a4009be0).stop 2021-11-19T08:45:01.312 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.326+0000 7f19b5343700 1 -- 172.21.15.23:0/1114571273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19b00f66a0 msgr2=0x7f19b006b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:01.313 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.326+0000 7f19b5343700 1 --2- 172.21.15.23:0/1114571273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19b00f66a0 0x7f19b006b7c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f19a0004e30 tx=0x7f19a0009b30).stop 2021-11-19T08:45:01.313 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.326+0000 7f19b5343700 1 -- 172.21.15.23:0/1114571273 shutdown_connections 2021-11-19T08:45:01.313 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.326+0000 7f19b5343700 1 --2- 172.21.15.23:0/1114571273 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f199805c2a0 0x7f199805e760 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:01.313 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.326+0000 7f19b5343700 1 --2- 172.21.15.23:0/1114571273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19b00f66a0 0x7f19b006b7c0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:01.314 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.326+0000 7f19b5343700 1 --2- 172.21.15.23:0/1114571273 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19b00f7940 0x7f19b0069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:01.314 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.326+0000 7f19b5343700 1 -- 172.21.15.23:0/1114571273 >> 172.21.15.23:0/1114571273 conn(0x7f19b00f1da0 msgr2=0x7f19b00fb390 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:01.315 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.327+0000 7f19b5343700 1 -- 172.21.15.23:0/1114571273 shutdown_connections 2021-11-19T08:45:01.315 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:01.327+0000 7f19b5343700 1 -- 172.21.15.23:0/1114571273 wait complete. 2021-11-19T08:45:01.315 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 93 2021-11-19T08:45:01.323 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:58.255999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:01.324 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:rejoin 2021-11-19T08:45:01.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:01 smithi023 conmon[33490]: audit 2021-11-19T08:45:01.325696+0000 mon.smithi023 (mon. 2021-11-19T08:45:01.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:01 smithi023 conmon[33490]: 0) 901 : audit [DBG] from='client.? 172.21.15.23:0/1114571273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:01.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:01 smithi134 conmon[36363]: audit 2021-11-19T08:45:01.325696+0000 mon.smithi023 (mon.0) 901 : audit [DBG] from='client.? 172.21.15.23:0/1114571273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:02.325 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:02.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.469+0000 7f91b8644700 1 -- 172.21.15.23:0/680450088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91b00f6820 msgr2=0x7f91b00f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:02.456 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.469+0000 7f91b8644700 1 --2- 172.21.15.23:0/680450088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91b00f6820 0x7f91b00f6c80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f91a4004660 tx=0x7f91a4009b30).stop 2021-11-19T08:45:02.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.469+0000 7f91b8644700 1 -- 172.21.15.23:0/680450088 shutdown_connections 2021-11-19T08:45:02.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.469+0000 7f91b8644700 1 --2- 172.21.15.23:0/680450088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91b00f6820 0x7f91b00f6c80 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:02.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.469+0000 7f91b8644700 1 --2- 172.21.15.23:0/680450088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f91b00f5580 0x7f91b00f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:02.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.469+0000 7f91b8644700 1 -- 172.21.15.23:0/680450088 >> 172.21.15.23:0/680450088 conn(0x7f91b00f0ce0 msgr2=0x7f91b00f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:02.457 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.469+0000 7f91b8644700 1 -- 172.21.15.23:0/680450088 shutdown_connections 2021-11-19T08:45:02.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.469+0000 7f91b8644700 1 -- 172.21.15.23:0/680450088 wait complete. 2021-11-19T08:45:02.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b8644700 1 Processor -- start 2021-11-19T08:45:02.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b8644700 1 -- start start 2021-11-19T08:45:02.458 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b8644700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f91b00f5580 0x7f91b0106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:02.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b8644700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91b00f6820 0x7f91b0107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:02.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b8644700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b0107940 con 0x7f91b00f6820 2021-11-19T08:45:02.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b8644700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b0107a80 con 0x7f91b00f5580 2021-11-19T08:45:02.459 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b5bdf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91b00f6820 0x7f91b0107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:02.460 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b5bdf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91b00f6820 0x7f91b0107390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48786/0 (socket says 172.21.15.23:48786) 2021-11-19T08:45:02.460 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b5bdf700 1 -- 172.21.15.23:0/387472227 learned_addr learned my addr 172.21.15.23:0/387472227 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:02.460 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b63e0700 1 --2- 172.21.15.23:0/387472227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f91b00f5580 0x7f91b0106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:02.460 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b5bdf700 1 -- 172.21.15.23:0/387472227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f91b00f5580 msgr2=0x7f91b0106ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:02.460 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b5bdf700 1 --2- 172.21.15.23:0/387472227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f91b00f5580 0x7f91b0106ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:02.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b5bdf700 1 -- 172.21.15.23:0/387472227 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a4005040 con 0x7f91b00f6820 2021-11-19T08:45:02.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b63e0700 1 --2- 172.21.15.23:0/387472227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f91b00f5580 0x7f91b0106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:45:02.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91b5bdf700 1 --2- 172.21.15.23:0/387472227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91b00f6820 0x7f91b0107390 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f91a4009f50 tx=0x7f91a4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:02.461 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.470+0000 7f91aeffd700 1 -- 172.21.15.23:0/387472227 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91a4018070 con 0x7f91b00f6820 2021-11-19T08:45:02.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.471+0000 7f91aeffd700 1 -- 172.21.15.23:0/387472227 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91a4013410 con 0x7f91b00f6820 2021-11-19T08:45:02.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.471+0000 7f91b8644700 1 -- 172.21.15.23:0/387472227 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b010c4c0 con 0x7f91b00f6820 2021-11-19T08:45:02.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.471+0000 7f91aeffd700 1 -- 172.21.15.23:0/387472227 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91a401c3c0 con 0x7f91b00f6820 2021-11-19T08:45:02.462 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.471+0000 7f91b8644700 1 -- 172.21.15.23:0/387472227 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b010ca80 con 0x7f91b00f6820 2021-11-19T08:45:02.463 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.472+0000 7f91b8644700 1 -- 172.21.15.23:0/387472227 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b0027a00 con 0x7f91b00f6820 2021-11-19T08:45:02.464 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.472+0000 7f91aeffd700 1 -- 172.21.15.23:0/387472227 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f91a401d420 con 0x7f91b00f6820 2021-11-19T08:45:02.464 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.472+0000 7f91aeffd700 1 --2- 172.21.15.23:0/387472227 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f919805c2a0 0x7f919805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:02.464 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.472+0000 7f91aeffd700 1 -- 172.21.15.23:0/387472227 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f91a4078120 con 0x7f91b00f6820 2021-11-19T08:45:02.464 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.472+0000 7f91b63e0700 1 --2- 172.21.15.23:0/387472227 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f919805c2a0 0x7f919805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:02.465 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.473+0000 7f91b63e0700 1 --2- 172.21.15.23:0/387472227 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f919805c2a0 0x7f919805e760 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f91b00f6680 tx=0x7f91a800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:02.465 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.476+0000 7f91aeffd700 1 -- 172.21.15.23:0/387472227 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f91a404a880 con 0x7f91b00f6820 2021-11-19T08:45:02.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:02 smithi023 conmon[33490]: cluster 2021-11-19T08:45:00.898841+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:45:02.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:02 smithi023 conmon[33490]: mgr.14164) 330 : cluster [DBG] pgmap v307: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 307 B/s wr, 1 op/s 2021-11-19T08:45:02.655 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.671+0000 7f91b8644700 1 -- 172.21.15.23:0/387472227 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91b00feec0 con 0x7f91b00f6820 2021-11-19T08:45:02.656 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.671+0000 7f91aeffd700 1 -- 172.21.15.23:0/387472227 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f91a404dea0 con 0x7f91b00f6820 2021-11-19T08:45:02.656 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:02.657 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78},{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:58.255999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.673+0000 7f91b8644700 1 -- 172.21.15.23:0/387472227 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f919805c2a0 msgr2=0x7f919805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.673+0000 7f91b8644700 1 --2- 172.21.15.23:0/387472227 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f919805c2a0 0x7f919805e760 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f91b00f6680 tx=0x7f91a800b040).stop 2021-11-19T08:45:02.658 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.673+0000 7f91b8644700 1 -- 172.21.15.23:0/387472227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91b00f6820 msgr2=0x7f91b0107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:02.659 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.673+0000 7f91b8644700 1 --2- 172.21.15.23:0/387472227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91b00f6820 0x7f91b0107390 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f91a4009f50 tx=0x7f91a4009b30).stop 2021-11-19T08:45:02.659 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.673+0000 7f91b8644700 1 -- 172.21.15.23:0/387472227 shutdown_connections 2021-11-19T08:45:02.659 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.673+0000 7f91b8644700 1 --2- 172.21.15.23:0/387472227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f91b00f5580 0x7f91b0106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:02.659 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.673+0000 7f91b8644700 1 --2- 172.21.15.23:0/387472227 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f919805c2a0 0x7f919805e760 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:02.660 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.673+0000 7f91b8644700 1 --2- 172.21.15.23:0/387472227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91b00f6820 0x7f91b0107390 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:02.660 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.673+0000 7f91b8644700 1 -- 172.21.15.23:0/387472227 >> 172.21.15.23:0/387472227 conn(0x7f91b00f0ce0 msgr2=0x7f91b00fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:02.660 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.673+0000 7f91b8644700 1 -- 172.21.15.23:0/387472227 shutdown_connections 2021-11-19T08:45:02.660 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:02.673+0000 7f91b8644700 1 -- 172.21.15.23:0/387472227 wait complete. 2021-11-19T08:45:02.660 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 93 2021-11-19T08:45:02.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:58.255999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:02.670 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:rejoin 2021-11-19T08:45:02.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:02 smithi134 conmon[36363]: cluster 2021-11-19T08:45:00.898841+0000 mgr.smithi023.ehlpcl (mgr.14164) 330 : cluster 2021-11-19T08:45:02.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:02 smithi134 conmon[36363]: [DBG] pgmap v307: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 307 B/s wr, 1 op/s 2021-11-19T08:45:03.672 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:03.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:03 smithi134 conmon[36363]: audit 2021-11-19T08:45:02.672552+0000 mon.smithi023 (mon.0) 902 : audit [DBG] from='client.? 172.21.15.23:0/387472227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:03.806 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.820+0000 7f27324f2700 1 -- 172.21.15.23:0/3215884151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f272c0f55a0 msgr2=0x7f272c0f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:03.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.820+0000 7f27324f2700 1 --2- 172.21.15.23:0/3215884151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f272c0f55a0 0x7f272c0f59c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f271c004660 tx=0x7f271c009b30).stop 2021-11-19T08:45:03.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.820+0000 7f27324f2700 1 -- 172.21.15.23:0/3215884151 shutdown_connections 2021-11-19T08:45:03.807 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.820+0000 7f27324f2700 1 --2- 172.21.15.23:0/3215884151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f272c0f6840 0x7f272c0f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:03.808 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.820+0000 7f27324f2700 1 --2- 172.21.15.23:0/3215884151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f272c0f55a0 0x7f272c0f59c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:03.808 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.820+0000 7f27324f2700 1 -- 172.21.15.23:0/3215884151 >> 172.21.15.23:0/3215884151 conn(0x7f272c0f0ce0 msgr2=0x7f272c0f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:03.808 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.820+0000 7f27324f2700 1 -- 172.21.15.23:0/3215884151 shutdown_connections 2021-11-19T08:45:03.808 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.820+0000 7f27324f2700 1 -- 172.21.15.23:0/3215884151 wait complete. 2021-11-19T08:45:03.809 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.820+0000 7f27324f2700 1 Processor -- start 2021-11-19T08:45:03.809 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.821+0000 7f27324f2700 1 -- start start 2021-11-19T08:45:03.809 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.821+0000 7f27324f2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f272c0f55a0 0x7f272c106e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:03.809 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.821+0000 7f27324f2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f272c0f6840 0x7f272c107330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:03.810 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.821+0000 7f27324f2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272c1078e0 con 0x7f272c0f55a0 2021-11-19T08:45:03.810 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.821+0000 7f27324f2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272c107a20 con 0x7f272c0f6840 2021-11-19T08:45:03.810 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.821+0000 7f272bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f272c0f55a0 0x7f272c106e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:03.811 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.821+0000 7f272bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f272c0f55a0 0x7f272c106e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48794/0 (socket says 172.21.15.23:48794) 2021-11-19T08:45:03.811 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.821+0000 7f272bfff700 1 -- 172.21.15.23:0/1792677048 learned_addr learned my addr 172.21.15.23:0/1792677048 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:03.811 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.821+0000 7f272bfff700 1 -- 172.21.15.23:0/1792677048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f272c0f6840 msgr2=0x7f272c107330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:03.811 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.821+0000 7f272bfff700 1 --2- 172.21.15.23:0/1792677048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f272c0f6840 0x7f272c107330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:45:03.812 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.821+0000 7f272bfff700 1 -- 172.21.15.23:0/1792677048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c005040 con 0x7f272c0f55a0 2021-11-19T08:45:03.812 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.821+0000 7f272bfff700 1 --2- 172.21.15.23:0/1792677048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f272c0f55a0 0x7f272c106e40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f271c004e30 tx=0x7f271c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:03.812 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.822+0000 7f2728ff9700 1 -- 172.21.15.23:0/1792677048 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f271c018070 con 0x7f272c0f55a0 2021-11-19T08:45:03.812 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.822+0000 7f2728ff9700 1 -- 172.21.15.23:0/1792677048 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f271c013410 con 0x7f272c0f55a0 2021-11-19T08:45:03.812 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.822+0000 7f27324f2700 1 -- 172.21.15.23:0/1792677048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272c10c460 con 0x7f272c0f55a0 2021-11-19T08:45:03.813 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.822+0000 7f2728ff9700 1 -- 172.21.15.23:0/1792677048 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f271c01c3c0 con 0x7f272c0f55a0 2021-11-19T08:45:03.813 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.822+0000 7f27324f2700 1 -- 172.21.15.23:0/1792677048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f272c10ca20 con 0x7f272c0f55a0 2021-11-19T08:45:03.814 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.823+0000 7f27324f2700 1 -- 172.21.15.23:0/1792677048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272c0fdd30 con 0x7f272c0f55a0 2021-11-19T08:45:03.815 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.823+0000 7f2728ff9700 1 -- 172.21.15.23:0/1792677048 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f271c01d420 con 0x7f272c0f55a0 2021-11-19T08:45:03.815 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.823+0000 7f2728ff9700 1 --2- 172.21.15.23:0/1792677048 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f271405c2a0 0x7f271405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:03.815 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.823+0000 7f2728ff9700 1 -- 172.21.15.23:0/1792677048 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f271c078020 con 0x7f272c0f55a0 2021-11-19T08:45:03.815 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.823+0000 7f272b7fe700 1 --2- 172.21.15.23:0/1792677048 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f271405c2a0 0x7f271405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:03.816 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.824+0000 7f272b7fe700 1 --2- 172.21.15.23:0/1792677048 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f271405c2a0 0x7f271405e760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f2720009880 tx=0x7f2720009a60).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:03.816 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:03.827+0000 7f2728ff9700 1 -- 172.21.15.23:0/1792677048 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f271c04a780 con 0x7f272c0f55a0 2021-11-19T08:45:03.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:03 smithi023 conmon[33490]: audit 2021-11-19T08:45:02.672552+0000 mon.smithi023 (mon.0) 902 2021-11-19T08:45:03.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:03 smithi023 conmon[33490]: : audit [DBG] from='client.? 172.21.15.23:0/387472227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:04.008 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:04.023+0000 7f27324f2700 1 -- 172.21.15.23:0/1792677048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f272c10d3e0 con 0x7f272c0f55a0 2021-11-19T08:45:04.009 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:04.024+0000 7f2728ff9700 1 -- 172.21.15.23:0/1792677048 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f271c04dda0 con 0x7f272c0f55a0 2021-11-19T08:45:04.009 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:04.009 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78},{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:58.255999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:04.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:04.026+0000 7f27324f2700 1 -- 172.21.15.23:0/1792677048 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f271405c2a0 msgr2=0x7f271405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:04.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:04.026+0000 7f27324f2700 1 --2- 172.21.15.23:0/1792677048 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f271405c2a0 0x7f271405e760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f2720009880 tx=0x7f2720009a60).stop 2021-11-19T08:45:04.011 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:04.026+0000 7f27324f2700 1 -- 172.21.15.23:0/1792677048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f272c0f55a0 msgr2=0x7f272c106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:04.026+0000 7f27324f2700 1 --2- 172.21.15.23:0/1792677048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f272c0f55a0 0x7f272c106e40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f271c004e30 tx=0x7f271c009b30).stop 2021-11-19T08:45:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:04.026+0000 7f27324f2700 1 -- 172.21.15.23:0/1792677048 shutdown_connections 2021-11-19T08:45:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:04.026+0000 7f27324f2700 1 --2- 172.21.15.23:0/1792677048 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f271405c2a0 0x7f271405e760 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:04.026+0000 7f27324f2700 1 --2- 172.21.15.23:0/1792677048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f272c0f55a0 0x7f272c106e40 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:04.012 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:04.026+0000 7f27324f2700 1 --2- 172.21.15.23:0/1792677048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f272c0f6840 0x7f272c107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-11-19T08:45:04.013 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:04.026+0000 7f27324f2700 1 -- 172.21.15.23:0/1792677048 >> 172.21.15.23:0/1792677048 conn(0x7f272c0f0ce0 msgr2=0x7f272c05f450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:04.013 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:04.026+0000 7f27324f2700 1 -- 172.21.15.23:0/1792677048 shutdown_connections 2021-11-19T08:45:04.013 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:04.026+0000 7f27324f2700 1 -- 172.21.15.23:0/1792677048 wait complete. 2021-11-19T08:45:04.013 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 93 2021-11-19T08:45:04.023 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:58.255999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:04.023 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:rejoin 2021-11-19T08:45:04.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:04 smithi134 conmon[36363]: cluster 2021-11-19T08:45:02.899434+0000 mgr.smithi023.ehlpcl (mgr.14164) 331 : cluster [DBG] pgmap v308: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-19T08:45:04.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:04 smithi134 conmon[36363]: audit 2021-11-19T08:45 2021-11-19T08:45:04.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:04 smithi134 conmon[36363]: :04.025126+0000 mon.smithi023 (mon.0) 903 : audit [DBG] from='client.? 172.21.15.23:0/1792677048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:04.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:04 smithi023 conmon[33490]: cluster 2021-11-19T08:45:02.899434+0000 mgr.smithi023.ehlpcl (mgr.14164) 331 : cluster [DBG] pgmap v308: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 409 B/s wr, 1 op/s 2021-11-19T08:45:04.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:04 smithi023 conmon[33490]: audit 2021-11-19T08:45:04.025126+0000 mon.smithi023 (mon.0) 903 : audit [DBG] from='client.? 172.21.15.23:0/1792677048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:05.025 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.173+0000 7fe6e4d65700 1 -- 172.21.15.23:0/1285079241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e00f4bf0 msgr2=0x7fe6e00f5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.173+0000 7fe6e4d65700 1 --2- 172.21.15.23:0/1285079241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e00f4bf0 0x7fe6e00f5010 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7fe6d00099e0 tx=0x7fe6d0004d10).stop 2021-11-19T08:45:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.173+0000 7fe6e4d65700 1 -- 172.21.15.23:0/1285079241 shutdown_connections 2021-11-19T08:45:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.173+0000 7fe6e4d65700 1 --2- 172.21.15.23:0/1285079241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6e00f5e90 0x7fe6e00f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.173+0000 7fe6e4d65700 1 --2- 172.21.15.23:0/1285079241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e00f4bf0 0x7fe6e00f5010 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.173+0000 7fe6e4d65700 1 -- 172.21.15.23:0/1285079241 >> 172.21.15.23:0/1285079241 conn(0x7fe6e00f0330 msgr2=0x7fe6e00f2770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.173+0000 7fe6e4d65700 1 -- 172.21.15.23:0/1285079241 shutdown_connections 2021-11-19T08:45:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.173+0000 7fe6e4d65700 1 -- 172.21.15.23:0/1285079241 wait complete. 2021-11-19T08:45:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.173+0000 7fe6e4d65700 1 Processor -- start 2021-11-19T08:45:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.173+0000 7fe6e4d65700 1 -- start start 2021-11-19T08:45:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6e4d65700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e00f4bf0 0x7fe6e0106e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6e4d65700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6e00f5e90 0x7fe6e0107370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6e4d65700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e0107920 con 0x7fe6e00f4bf0 2021-11-19T08:45:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6e4d65700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e0107a60 con 0x7fe6e00f5e90 2021-11-19T08:45:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6df7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e00f4bf0 0x7fe6e0106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6df7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e00f4bf0 0x7fe6e0106e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48806/0 (socket says 172.21.15.23:48806) 2021-11-19T08:45:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6df7fe700 1 -- 172.21.15.23:0/925013793 learned_addr learned my addr 172.21.15.23:0/925013793 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6deffd700 1 --2- 172.21.15.23:0/925013793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6e00f5e90 0x7fe6e0107370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6df7fe700 1 -- 172.21.15.23:0/925013793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6e00f5e90 msgr2=0x7fe6e0107370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6df7fe700 1 --2- 172.21.15.23:0/925013793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6e00f5e90 0x7fe6e0107370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6df7fe700 1 -- 172.21.15.23:0/925013793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d0009640 con 0x7fe6e00f4bf0 2021-11-19T08:45:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6df7fe700 1 --2- 172.21.15.23:0/925013793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e00f4bf0 0x7fe6e0106e80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fe6d00078d0 tx=0x7fe6d0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6d7fff700 1 -- 172.21.15.23:0/925013793 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6d000e070 con 0x7fe6e00f4bf0 2021-11-19T08:45:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6e4d65700 1 -- 172.21.15.23:0/925013793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e01083a0 con 0x7fe6e00f4bf0 2021-11-19T08:45:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6d7fff700 1 -- 172.21.15.23:0/925013793 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe6d001d8d0 con 0x7fe6e00f4bf0 2021-11-19T08:45:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6d7fff700 1 -- 172.21.15.23:0/925013793 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6d0018070 con 0x7fe6e00f4bf0 2021-11-19T08:45:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.174+0000 7fe6e4d65700 1 -- 172.21.15.23:0/925013793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e0108900 con 0x7fe6e00f4bf0 2021-11-19T08:45:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.175+0000 7fe6e4d65700 1 -- 172.21.15.23:0/925013793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e0045ad0 con 0x7fe6e00f4bf0 2021-11-19T08:45:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.176+0000 7fe6d7fff700 1 -- 172.21.15.23:0/925013793 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe6d0024060 con 0x7fe6e00f4bf0 2021-11-19T08:45:05.168 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.176+0000 7fe6d7fff700 1 --2- 172.21.15.23:0/925013793 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe6cc05c300 0x7fe6cc05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:05.168 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.176+0000 7fe6d7fff700 1 -- 172.21.15.23:0/925013793 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fe6d0077ba0 con 0x7fe6e00f4bf0 2021-11-19T08:45:05.168 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.176+0000 7fe6deffd700 1 --2- 172.21.15.23:0/925013793 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe6cc05c300 0x7fe6cc05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:05.168 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.177+0000 7fe6deffd700 1 --2- 172.21.15.23:0/925013793 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe6cc05c300 0x7fe6cc05e7c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fe6c8009da0 tx=0x7fe6c800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:05.169 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.180+0000 7fe6d7fff700 1 -- 172.21.15.23:0/925013793 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe6d004a300 con 0x7fe6e00f4bf0 2021-11-19T08:45:05.358 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.374+0000 7fe6e4d65700 1 -- 172.21.15.23:0/925013793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6e00fd4c0 con 0x7fe6e00f4bf0 2021-11-19T08:45:05.359 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.374+0000 7fe6d7fff700 1 -- 172.21.15.23:0/925013793 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2951 (secure 0 0 0) 0x7fe6d004d920 con 0x7fe6e00f4bf0 2021-11-19T08:45:05.360 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:05.360 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78},{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:58.255999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:05.362 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.376+0000 7fe6e4d65700 1 -- 172.21.15.23:0/925013793 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe6cc05c300 msgr2=0x7fe6cc05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:05.362 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.376+0000 7fe6e4d65700 1 --2- 172.21.15.23:0/925013793 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe6cc05c300 0x7fe6cc05e7c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fe6c8009da0 tx=0x7fe6c800b040).stop 2021-11-19T08:45:05.362 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.376+0000 7fe6e4d65700 1 -- 172.21.15.23:0/925013793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e00f4bf0 msgr2=0x7fe6e0106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:05.362 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.376+0000 7fe6e4d65700 1 --2- 172.21.15.23:0/925013793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e00f4bf0 0x7fe6e0106e80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fe6d00078d0 tx=0x7fe6d0004d10).stop 2021-11-19T08:45:05.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.376+0000 7fe6e4d65700 1 -- 172.21.15.23:0/925013793 shutdown_connections 2021-11-19T08:45:05.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.376+0000 7fe6e4d65700 1 --2- 172.21.15.23:0/925013793 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe6cc05c300 0x7fe6cc05e7c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:05.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.376+0000 7fe6e4d65700 1 --2- 172.21.15.23:0/925013793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e00f4bf0 0x7fe6e0106e80 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:05.363 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.376+0000 7fe6e4d65700 1 --2- 172.21.15.23:0/925013793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6e00f5e90 0x7fe6e0107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:05.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.376+0000 7fe6e4d65700 1 -- 172.21.15.23:0/925013793 >> 172.21.15.23:0/925013793 conn(0x7fe6e00f0330 msgr2=0x7fe6e00fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:05.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.376+0000 7fe6e4d65700 1 -- 172.21.15.23:0/925013793 shutdown_connections 2021-11-19T08:45:05.364 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:05.376+0000 7fe6e4d65700 1 -- 172.21.15.23:0/925013793 wait complete. 2021-11-19T08:45:05.364 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 93 2021-11-19T08:45:05.373 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:58.255999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:05.374 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:rejoin 2021-11-19T08:45:05.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:05 smithi023 conmon[33490]: audit 2021-11-19T08:45:05.375514+0000 mon.smithi023 (mon.0) 904 : audit [DBG] 2021-11-19T08:45:05.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:05 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/925013793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:06.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:05 smithi134 conmon[36363]: audit 2021-11-19T08:45:05.375514+0000 mon.smithi023 (mon.0) 904 : audit [DBG] from='client.? 172.21.15.23:0/925013793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:06.375 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:06.512 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.522+0000 7fd8ad876700 1 -- 172.21.15.23:0/3086923713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8a80f6820 msgr2=0x7fd8a80f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:06.513 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.522+0000 7fd8ad876700 1 --2- 172.21.15.23:0/3086923713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8a80f6820 0x7fd8a80f6c80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7fd890004660 tx=0x7fd890009b30).stop 2021-11-19T08:45:06.513 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.522+0000 7fd8ad876700 1 -- 172.21.15.23:0/3086923713 shutdown_connections 2021-11-19T08:45:06.514 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.522+0000 7fd8ad876700 1 --2- 172.21.15.23:0/3086923713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8a80f6820 0x7fd8a80f6c80 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:06.515 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.522+0000 7fd8ad876700 1 --2- 172.21.15.23:0/3086923713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a80f5580 0x7fd8a80f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:06.515 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.522+0000 7fd8ad876700 1 -- 172.21.15.23:0/3086923713 >> 172.21.15.23:0/3086923713 conn(0x7fd8a80f0ce0 msgr2=0x7fd8a80f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:06.515 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.522+0000 7fd8ad876700 1 -- 172.21.15.23:0/3086923713 shutdown_connections 2021-11-19T08:45:06.515 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.522+0000 7fd8ad876700 1 -- 172.21.15.23:0/3086923713 wait complete. 2021-11-19T08:45:06.516 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.523+0000 7fd8ad876700 1 Processor -- start 2021-11-19T08:45:06.516 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.523+0000 7fd8ad876700 1 -- start start 2021-11-19T08:45:06.516 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.523+0000 7fd8ad876700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a80f5580 0x7fd8a8060550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:06.517 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.523+0000 7fd8ad876700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8a80f6820 0x7fd8a810bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:06.517 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.523+0000 7fd8ad876700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a810c1c0 con 0x7fd8a80f6820 2021-11-19T08:45:06.517 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.523+0000 7fd8ad876700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a810c300 con 0x7fd8a80f5580 2021-11-19T08:45:06.518 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.523+0000 7fd8a67fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8a80f6820 0x7fd8a810bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:06.518 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.523+0000 7fd8a67fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8a80f6820 0x7fd8a810bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48820/0 (socket says 172.21.15.23:48820) 2021-11-19T08:45:06.518 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.523+0000 7fd8a67fc700 1 -- 172.21.15.23:0/374475557 learned_addr learned my addr 172.21.15.23:0/374475557 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:06.519 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.523+0000 7fd8a6ffd700 1 --2- 172.21.15.23:0/374475557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a80f5580 0x7fd8a8060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:06.519 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.523+0000 7fd8a67fc700 1 -- 172.21.15.23:0/374475557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a80f5580 msgr2=0x7fd8a8060550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:06.519 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.523+0000 7fd8a67fc700 1 --2- 172.21.15.23:0/374475557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a80f5580 0x7fd8a8060550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:06.520 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.523+0000 7fd8a67fc700 1 -- 172.21.15.23:0/374475557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd890005040 con 0x7fd8a80f6820 2021-11-19T08:45:06.520 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.524+0000 7fd8a67fc700 1 --2- 172.21.15.23:0/374475557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8a80f6820 0x7fd8a810bcd0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fd890009f50 tx=0x7fd890009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:06.520 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.524+0000 7fd89f7fe700 1 -- 172.21.15.23:0/374475557 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd890018070 con 0x7fd8a80f6820 2021-11-19T08:45:06.521 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.524+0000 7fd89f7fe700 1 -- 172.21.15.23:0/374475557 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd890013410 con 0x7fd8a80f6820 2021-11-19T08:45:06.521 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.524+0000 7fd8ad876700 1 -- 172.21.15.23:0/374475557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a810c500 con 0x7fd8a80f6820 2021-11-19T08:45:06.521 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.524+0000 7fd89f7fe700 1 -- 172.21.15.23:0/374475557 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd89001c3c0 con 0x7fd8a80f6820 2021-11-19T08:45:06.521 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.524+0000 7fd8ad876700 1 -- 172.21.15.23:0/374475557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a810ca30 con 0x7fd8a80f6820 2021-11-19T08:45:06.521 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.525+0000 7fd8ad876700 1 -- 172.21.15.23:0/374475557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a8046330 con 0x7fd8a80f6820 2021-11-19T08:45:06.522 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.525+0000 7fd89f7fe700 1 -- 172.21.15.23:0/374475557 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd89001d420 con 0x7fd8a80f6820 2021-11-19T08:45:06.522 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.525+0000 7fd89f7fe700 1 --2- 172.21.15.23:0/374475557 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd89405c2a0 0x7fd89405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:06.522 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.525+0000 7fd89f7fe700 1 -- 172.21.15.23:0/374475557 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fd890077ff0 con 0x7fd8a80f6820 2021-11-19T08:45:06.522 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.526+0000 7fd8a6ffd700 1 --2- 172.21.15.23:0/374475557 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd89405c2a0 0x7fd89405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:06.523 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.526+0000 7fd8a6ffd700 1 --2- 172.21.15.23:0/374475557 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd89405c2a0 0x7fd89405e760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fd8a80f6680 tx=0x7fd89800b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:06.523 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.529+0000 7fd89f7fe700 1 -- 172.21.15.23:0/374475557 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd89004a7d0 con 0x7fd8a80f6820 2021-11-19T08:45:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.725+0000 7fd8ad876700 1 -- 172.21.15.23:0/374475557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8a8027a00 con 0x7fd8a80f6820 2021-11-19T08:45:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.725+0000 7fd89f7fe700 1 -- 172.21.15.23:0/374475557 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2951 (secure 0 0 0) 0x7fd89004ddf0 con 0x7fd8a80f6820 2021-11-19T08:45:06.710 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:06.711 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78},{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:58.255999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.727+0000 7fd8ad876700 1 -- 172.21.15.23:0/374475557 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd89405c2a0 msgr2=0x7fd89405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.727+0000 7fd8ad876700 1 --2- 172.21.15.23:0/374475557 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd89405c2a0 0x7fd89405e760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fd8a80f6680 tx=0x7fd89800b040).stop 2021-11-19T08:45:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.727+0000 7fd8ad876700 1 -- 172.21.15.23:0/374475557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8a80f6820 msgr2=0x7fd8a810bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.727+0000 7fd8ad876700 1 --2- 172.21.15.23:0/374475557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8a80f6820 0x7fd8a810bcd0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fd890009f50 tx=0x7fd890009b30).stop 2021-11-19T08:45:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.727+0000 7fd8ad876700 1 -- 172.21.15.23:0/374475557 shutdown_connections 2021-11-19T08:45:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.727+0000 7fd8ad876700 1 --2- 172.21.15.23:0/374475557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a80f5580 0x7fd8a8060550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.727+0000 7fd8ad876700 1 --2- 172.21.15.23:0/374475557 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fd89405c2a0 0x7fd89405e760 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.727+0000 7fd8ad876700 1 --2- 172.21.15.23:0/374475557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8a80f6820 0x7fd8a810bcd0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.727+0000 7fd8ad876700 1 -- 172.21.15.23:0/374475557 >> 172.21.15.23:0/374475557 conn(0x7fd8a80f0ce0 msgr2=0x7fd8a80fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.727+0000 7fd8ad876700 1 -- 172.21.15.23:0/374475557 shutdown_connections 2021-11-19T08:45:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:06.727+0000 7fd8ad876700 1 -- 172.21.15.23:0/374475557 wait complete. 2021-11-19T08:45:06.715 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 93 2021-11-19T08:45:06.722 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:06 smithi023 conmon[33490]: cluster 2021-11-19T08:45:04.899945+0000 mgr.smithi023.ehlpcl (mgr. 2021-11-19T08:45:06.722 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:06 smithi023 conmon[33490]: 14164) 332 : cluster [DBG] pgmap v309: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 384 B/s wr, 1 op/s 2021-11-19T08:45:06.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:58.255999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:06.724 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:rejoin 2021-11-19T08:45:07.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:06 smithi134 conmon[36363]: cluster 2021-11-19T08:45:04.899945+0000 mgr.smithi023.ehlpcl (mgr.14164) 332 : cluster [DBG] pgmap v309: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 384 B/s wr, 1 op/s 2021-11-19T08:45:07.725 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:07.866 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.873+0000 7f7a986c2700 1 -- 172.21.15.23:0/731270697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a900f7960 msgr2=0x7f7a900f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:07.867 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.873+0000 7f7a986c2700 1 --2- 172.21.15.23:0/731270697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a900f7960 0x7f7a900f7dc0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f7a80004660 tx=0x7f7a80009b30).stop 2021-11-19T08:45:07.867 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.873+0000 7f7a986c2700 1 -- 172.21.15.23:0/731270697 shutdown_connections 2021-11-19T08:45:07.867 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.873+0000 7f7a986c2700 1 --2- 172.21.15.23:0/731270697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a900f7960 0x7f7a900f7dc0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:07.868 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.873+0000 7f7a986c2700 1 --2- 172.21.15.23:0/731270697 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a900f66c0 0x7f7a900f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:07.868 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.874+0000 7f7a986c2700 1 -- 172.21.15.23:0/731270697 >> 172.21.15.23:0/731270697 conn(0x7f7a900f1e40 msgr2=0x7f7a900f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:07.868 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.874+0000 7f7a986c2700 1 -- 172.21.15.23:0/731270697 shutdown_connections 2021-11-19T08:45:07.868 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.874+0000 7f7a986c2700 1 -- 172.21.15.23:0/731270697 wait complete. 2021-11-19T08:45:07.869 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.874+0000 7f7a986c2700 1 Processor -- start 2021-11-19T08:45:07.869 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.874+0000 7f7a986c2700 1 -- start start 2021-11-19T08:45:07.869 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.874+0000 7f7a986c2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a900f66c0 0x7f7a9006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:07.869 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.874+0000 7f7a986c2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a900f7960 0x7f7a90069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:07.870 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.874+0000 7f7a986c2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9006bd70 con 0x7f7a900f7960 2021-11-19T08:45:07.870 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.874+0000 7f7a986c2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9006a310 con 0x7f7a900f66c0 2021-11-19T08:45:07.870 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.874+0000 7f7a95c5d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a900f7960 0x7f7a90069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:07.870 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.874+0000 7f7a95c5d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a900f7960 0x7f7a90069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48830/0 (socket says 172.21.15.23:48830) 2021-11-19T08:45:07.871 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.874+0000 7f7a95c5d700 1 -- 172.21.15.23:0/85892920 learned_addr learned my addr 172.21.15.23:0/85892920 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:07.871 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.875+0000 7f7a9645e700 1 --2- 172.21.15.23:0/85892920 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a900f66c0 0x7f7a9006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:07.871 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.875+0000 7f7a95c5d700 1 -- 172.21.15.23:0/85892920 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a900f66c0 msgr2=0x7f7a9006b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:07.871 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.875+0000 7f7a95c5d700 1 --2- 172.21.15.23:0/85892920 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a900f66c0 0x7f7a9006b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:07.872 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.875+0000 7f7a95c5d700 1 -- 172.21.15.23:0/85892920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a80005040 con 0x7f7a900f7960 2021-11-19T08:45:07.872 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.875+0000 7f7a95c5d700 1 --2- 172.21.15.23:0/85892920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a900f7960 0x7f7a90069e20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f7a80009f50 tx=0x7f7a80009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:07.872 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.875+0000 7f7a86ffd700 1 -- 172.21.15.23:0/85892920 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a80018070 con 0x7f7a900f7960 2021-11-19T08:45:07.872 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.875+0000 7f7a86ffd700 1 -- 172.21.15.23:0/85892920 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a80013410 con 0x7f7a900f7960 2021-11-19T08:45:07.872 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.875+0000 7f7a986c2700 1 -- 172.21.15.23:0/85892920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a9006a530 con 0x7f7a900f7960 2021-11-19T08:45:07.873 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.875+0000 7f7a86ffd700 1 -- 172.21.15.23:0/85892920 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a8001c3c0 con 0x7f7a900f7960 2021-11-19T08:45:07.873 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.875+0000 7f7a986c2700 1 -- 172.21.15.23:0/85892920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9006aa90 con 0x7f7a900f7960 2021-11-19T08:45:07.873 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.876+0000 7f7a986c2700 1 -- 172.21.15.23:0/85892920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a90046330 con 0x7f7a900f7960 2021-11-19T08:45:07.873 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.877+0000 7f7a86ffd700 1 -- 172.21.15.23:0/85892920 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7a8001d420 con 0x7f7a900f7960 2021-11-19T08:45:07.874 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.877+0000 7f7a86ffd700 1 --2- 172.21.15.23:0/85892920 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f7a7c05c2a0 0x7f7a7c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:07.875 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.877+0000 7f7a86ffd700 1 -- 172.21.15.23:0/85892920 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f7a800781e0 con 0x7f7a900f7960 2021-11-19T08:45:07.875 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.877+0000 7f7a9645e700 1 --2- 172.21.15.23:0/85892920 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f7a7c05c2a0 0x7f7a7c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:07.875 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.877+0000 7f7a9645e700 1 --2- 172.21.15.23:0/85892920 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f7a7c05c2a0 0x7f7a7c05e760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f7a900f77c0 tx=0x7f7a8c00b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:07.875 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:07.881+0000 7f7a86ffd700 1 -- 172.21.15.23:0/85892920 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a8004a880 con 0x7f7a900f7960 2021-11-19T08:45:07.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:07 smithi023 conmon[33490]: audit 2021-11-19T08:45:06.726156+0000 mon.smithi023 (mon.0) 905 : audit [DBG] 2021-11-19T08:45:07.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:07 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/374475557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:08.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:07 smithi134 conmon[36363]: audit 2021-11-19T08:45:06.726156+0000 mon.smithi023 (mon.0) 905 : 2021-11-19T08:45:08.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:07 smithi134 conmon[36363]: audit [DBG] from='client.? 172.21.15.23:0/374475557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:08.078+0000 7f7a986c2700 1 -- 172.21.15.23:0/85892920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a90027a00 con 0x7f7a900f7960 2021-11-19T08:45:08.063 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:08.079+0000 7f7a86ffd700 1 -- 172.21.15.23:0/85892920 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f7a8004dea0 con 0x7f7a900f7960 2021-11-19T08:45:08.064 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:08.064 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78},{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:58.255999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:08.065 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:08.080+0000 7f7a986c2700 1 -- 172.21.15.23:0/85892920 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f7a7c05c2a0 msgr2=0x7f7a7c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:08.065 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:08.080+0000 7f7a986c2700 1 --2- 172.21.15.23:0/85892920 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f7a7c05c2a0 0x7f7a7c05e760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f7a900f77c0 tx=0x7f7a8c00b040).stop 2021-11-19T08:45:08.066 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:08.080+0000 7f7a986c2700 1 -- 172.21.15.23:0/85892920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a900f7960 msgr2=0x7f7a90069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:08.066 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:08.080+0000 7f7a986c2700 1 --2- 172.21.15.23:0/85892920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a900f7960 0x7f7a90069e20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f7a80009f50 tx=0x7f7a80009b30).stop 2021-11-19T08:45:08.066 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:08.080+0000 7f7a986c2700 1 -- 172.21.15.23:0/85892920 shutdown_connections 2021-11-19T08:45:08.066 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:08.080+0000 7f7a986c2700 1 --2- 172.21.15.23:0/85892920 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a900f66c0 0x7f7a9006b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:08.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:08.080+0000 7f7a986c2700 1 --2- 172.21.15.23:0/85892920 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f7a7c05c2a0 0x7f7a7c05e760 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:08.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:08.080+0000 7f7a986c2700 1 --2- 172.21.15.23:0/85892920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a900f7960 0x7f7a90069e20 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:08.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:08.080+0000 7f7a986c2700 1 -- 172.21.15.23:0/85892920 >> 172.21.15.23:0/85892920 conn(0x7f7a900f1e40 msgr2=0x7f7a900fb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:08.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:08.080+0000 7f7a986c2700 1 -- 172.21.15.23:0/85892920 shutdown_connections 2021-11-19T08:45:08.067 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:08.080+0000 7f7a986c2700 1 -- 172.21.15.23:0/85892920 wait complete. 2021-11-19T08:45:08.068 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 93 2021-11-19T08:45:08.077 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:58.255999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:08.077 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:rejoin 2021-11-19T08:45:08.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:08 smithi023 conmon[33490]: cluster 2021-11-19T08:45:06.900647+0000 mgr.smithi023.ehlpcl (mgr.14164) 333 : 2021-11-19T08:45:08.891 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:08 smithi023 conmon[33490]: cluster [DBG] pgmap v310: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 341 B/s wr, 0 op/s 2021-11-19T08:45:08.892 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:08 smithi023 conmon[33490]: audit 2021-11-19T08:45:08.079620+0000 mon.smithi023 (mon.0) 906 : audit [DBG] from='client.? 172.21.15.23:0/85892920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:09.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:08 smithi134 conmon[36363]: cluster 2021-11-19T08:45:06.900647+0000 mgr.smithi023.ehlpcl (mgr 2021-11-19T08:45:09.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:08 smithi134 conmon[36363]: .14164) 333 : cluster [DBG] pgmap v310: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 341 B/s wr, 0 op/s 2021-11-19T08:45:09.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:08 smithi134 conmon[36363]: audit 2021-11-19T08:45:08.079620+0000 mon.smithi023 (mon.0) 906 : audit [DBG] from='client.? 172.21.15.23:0/85892920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:09.078 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:09.219 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.230+0000 7fa088749700 1 -- 172.21.15.23:0/2885428304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0800f5b90 msgr2=0x7fa0800f5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:09.219 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.230+0000 7fa088749700 1 --2- 172.21.15.23:0/2885428304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0800f5b90 0x7fa0800f5fb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7fa07c004660 tx=0x7fa07c009b30).stop 2021-11-19T08:45:09.220 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.230+0000 7fa088749700 1 -- 172.21.15.23:0/2885428304 shutdown_connections 2021-11-19T08:45:09.220 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.230+0000 7fa088749700 1 --2- 172.21.15.23:0/2885428304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0800f6530 0x7fa0800f6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:09.220 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.230+0000 7fa088749700 1 --2- 172.21.15.23:0/2885428304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0800f5b90 0x7fa0800f5fb0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:09.220 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.230+0000 7fa088749700 1 -- 172.21.15.23:0/2885428304 >> 172.21.15.23:0/2885428304 conn(0x7fa0800f04a0 msgr2=0x7fa0800f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:09.220 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.231+0000 7fa088749700 1 -- 172.21.15.23:0/2885428304 shutdown_connections 2021-11-19T08:45:09.221 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.231+0000 7fa088749700 1 -- 172.21.15.23:0/2885428304 wait complete. 2021-11-19T08:45:09.221 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.231+0000 7fa088749700 1 Processor -- start 2021-11-19T08:45:09.221 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.231+0000 7fa088749700 1 -- start start 2021-11-19T08:45:09.221 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.231+0000 7fa088749700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0800f5b90 0x7fa0800f5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:09.222 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.231+0000 7fa088749700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0800f6530 0x7fa08010bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:09.222 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.231+0000 7fa088749700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08010c110 con 0x7fa0800f5b90 2021-11-19T08:45:09.222 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.231+0000 7fa088749700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08010c250 con 0x7fa0800f6530 2021-11-19T08:45:09.222 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.231+0000 7fa0864e5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0800f5b90 0x7fa0800f5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:09.223 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.231+0000 7fa0864e5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0800f5b90 0x7fa0800f5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48838/0 (socket says 172.21.15.23:48838) 2021-11-19T08:45:09.223 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.231+0000 7fa0864e5700 1 -- 172.21.15.23:0/3824574647 learned_addr learned my addr 172.21.15.23:0/3824574647 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:09.223 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.232+0000 7fa085ce4700 1 --2- 172.21.15.23:0/3824574647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0800f6530 0x7fa08010bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:09.223 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.232+0000 7fa0864e5700 1 -- 172.21.15.23:0/3824574647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0800f6530 msgr2=0x7fa08010bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:09.224 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.232+0000 7fa0864e5700 1 --2- 172.21.15.23:0/3824574647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0800f6530 0x7fa08010bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:09.224 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.232+0000 7fa0864e5700 1 -- 172.21.15.23:0/3824574647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07c005040 con 0x7fa0800f5b90 2021-11-19T08:45:09.224 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.232+0000 7fa0864e5700 1 --2- 172.21.15.23:0/3824574647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0800f5b90 0x7fa0800f5390 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7fa07c005010 tx=0x7fa07c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:09.225 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.232+0000 7fa076ffd700 1 -- 172.21.15.23:0/3824574647 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa07c019070 con 0x7fa0800f5b90 2021-11-19T08:45:09.225 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.232+0000 7fa088749700 1 -- 172.21.15.23:0/3824574647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa08010c450 con 0x7fa0800f5b90 2021-11-19T08:45:09.225 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.232+0000 7fa076ffd700 1 -- 172.21.15.23:0/3824574647 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa07c007e70 con 0x7fa0800f5b90 2021-11-19T08:45:09.225 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.232+0000 7fa076ffd700 1 -- 172.21.15.23:0/3824574647 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa07c014d10 con 0x7fa0800f5b90 2021-11-19T08:45:09.225 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.232+0000 7fa088749700 1 -- 172.21.15.23:0/3824574647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08010c9b0 con 0x7fa0800f5b90 2021-11-19T08:45:09.226 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.233+0000 7fa088749700 1 -- 172.21.15.23:0/3824574647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa080027a00 con 0x7fa0800f5b90 2021-11-19T08:45:09.226 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.233+0000 7fa076ffd700 1 -- 172.21.15.23:0/3824574647 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa07c025070 con 0x7fa0800f5b90 2021-11-19T08:45:09.226 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.234+0000 7fa076ffd700 1 --2- 172.21.15.23:0/3824574647 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa06c05c2a0 0x7fa06c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:09.226 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.234+0000 7fa076ffd700 1 -- 172.21.15.23:0/3824574647 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fa07c077400 con 0x7fa0800f5b90 2021-11-19T08:45:09.226 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.234+0000 7fa085ce4700 1 --2- 172.21.15.23:0/3824574647 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa06c05c2a0 0x7fa06c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:09.227 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.234+0000 7fa085ce4700 1 --2- 172.21.15.23:0/3824574647 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa06c05c2a0 0x7fa06c05e760 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fa070009fd0 tx=0x7fa07000b040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:09.227 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.238+0000 7fa076ffd700 1 -- 172.21.15.23:0/3824574647 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa07c049b60 con 0x7fa0800f5b90 2021-11-19T08:45:09.421 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.437+0000 7fa088749700 1 -- 172.21.15.23:0/3824574647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0800fdd50 con 0x7fa0800f5b90 2021-11-19T08:45:09.422 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.437+0000 7fa076ffd700 1 -- 172.21.15.23:0/3824574647 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2951 (secure 0 0 0) 0x7fa07c04d180 con 0x7fa0800f5b90 2021-11-19T08:45:09.423 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:09.423 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78},{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:58.255999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:09.424 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.439+0000 7fa088749700 1 -- 172.21.15.23:0/3824574647 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa06c05c2a0 msgr2=0x7fa06c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:09.424 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.439+0000 7fa088749700 1 --2- 172.21.15.23:0/3824574647 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa06c05c2a0 0x7fa06c05e760 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fa070009fd0 tx=0x7fa07000b040).stop 2021-11-19T08:45:09.425 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.439+0000 7fa088749700 1 -- 172.21.15.23:0/3824574647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0800f5b90 msgr2=0x7fa0800f5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:09.425 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.439+0000 7fa088749700 1 --2- 172.21.15.23:0/3824574647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0800f5b90 0x7fa0800f5390 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7fa07c005010 tx=0x7fa07c00b040).stop 2021-11-19T08:45:09.425 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.439+0000 7fa088749700 1 -- 172.21.15.23:0/3824574647 shutdown_connections 2021-11-19T08:45:09.425 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.439+0000 7fa088749700 1 --2- 172.21.15.23:0/3824574647 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fa06c05c2a0 0x7fa06c05e760 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:09.426 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.439+0000 7fa088749700 1 --2- 172.21.15.23:0/3824574647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0800f5b90 0x7fa0800f5390 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:09.426 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.439+0000 7fa088749700 1 --2- 172.21.15.23:0/3824574647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0800f6530 0x7fa08010bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:09.426 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.439+0000 7fa088749700 1 -- 172.21.15.23:0/3824574647 >> 172.21.15.23:0/3824574647 conn(0x7fa0800f04a0 msgr2=0x7fa0800fa430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:09.427 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.439+0000 7fa088749700 1 -- 172.21.15.23:0/3824574647 shutdown_connections 2021-11-19T08:45:09.427 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:09.439+0000 7fa088749700 1 -- 172.21.15.23:0/3824574647 wait complete. 2021-11-19T08:45:09.428 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 93 2021-11-19T08:45:09.436 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:58.255999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:09.436 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:rejoin 2021-11-19T08:45:10.437 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:10.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:10 smithi134 conmon[36363]: cluster 2021-11-19T08:45:08.901326+0000 mgr.smithi023.ehlpcl (mgr.14164) 334 : cluster [DBG] pgmap v311: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 341 B/s wr, 0 op/s 2021-11-19T08:45:10.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:10 smithi134 conmon[36363]: audit 2021-11-19T08:45:09.438536+0000 mon.smithi023 (mon.0) 907 : audit [DBG] from='client.? 172.21.15.23:0/3824574647' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.587+0000 7f62259a2700 1 -- 172.21.15.23:0/391982325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62200f5460 msgr2=0x7f62200f5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.587+0000 7f62259a2700 1 --2- 172.21.15.23:0/391982325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62200f5460 0x7f62200f5880 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f62180099e0 tx=0x7f6218004d10).stop 2021-11-19T08:45:10.574 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.587+0000 7f62259a2700 1 -- 172.21.15.23:0/391982325 shutdown_connections 2021-11-19T08:45:10.575 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.587+0000 7f62259a2700 1 --2- 172.21.15.23:0/391982325 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62200f6700 0x7f62200f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:10.575 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.587+0000 7f62259a2700 1 --2- 172.21.15.23:0/391982325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62200f5460 0x7f62200f5880 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:10.575 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.587+0000 7f62259a2700 1 -- 172.21.15.23:0/391982325 >> 172.21.15.23:0/391982325 conn(0x7f62200f0ba0 msgr2=0x7f62200f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:10.575 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.587+0000 7f62259a2700 1 -- 172.21.15.23:0/391982325 shutdown_connections 2021-11-19T08:45:10.576 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.587+0000 7f62259a2700 1 -- 172.21.15.23:0/391982325 wait complete. 2021-11-19T08:45:10.576 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.588+0000 7f62259a2700 1 Processor -- start 2021-11-19T08:45:10.576 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.588+0000 7f62259a2700 1 -- start start 2021-11-19T08:45:10.576 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.588+0000 7f62259a2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62200f5460 0x7f6220105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:10.576 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.588+0000 7f62259a2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62200f6700 0x7f6220106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:10.577 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.588+0000 7f62259a2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6220106890 con 0x7f62200f6700 2021-11-19T08:45:10.577 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.588+0000 7f62259a2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62201070d0 con 0x7f62200f5460 2021-11-19T08:45:10.577 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.588+0000 7f621ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62200f6700 0x7f6220106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:10.577 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.588+0000 7f621ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62200f6700 0x7f6220106290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48850/0 (socket says 172.21.15.23:48850) 2021-11-19T08:45:10.578 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.588+0000 7f621ffff700 1 -- 172.21.15.23:0/854502477 learned_addr learned my addr 172.21.15.23:0/854502477 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:10.578 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.589+0000 7f62249a0700 1 --2- 172.21.15.23:0/854502477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62200f5460 0x7f6220105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:10.578 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.589+0000 7f621ffff700 1 -- 172.21.15.23:0/854502477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62200f5460 msgr2=0x7f6220105da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:10.578 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.589+0000 7f621ffff700 1 --2- 172.21.15.23:0/854502477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62200f5460 0x7f6220105da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:10.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.589+0000 7f621ffff700 1 -- 172.21.15.23:0/854502477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6218009640 con 0x7f62200f6700 2021-11-19T08:45:10.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.589+0000 7f621ffff700 1 --2- 172.21.15.23:0/854502477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62200f6700 0x7f6220106290 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f6214002700 tx=0x7f6214009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:10.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.589+0000 7f621d7fa700 1 -- 172.21.15.23:0/854502477 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6214009320 con 0x7f62200f6700 2021-11-19T08:45:10.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.589+0000 7f62259a2700 1 -- 172.21.15.23:0/854502477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62201073b0 con 0x7f62200f6700 2021-11-19T08:45:10.579 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.589+0000 7f621d7fa700 1 -- 172.21.15.23:0/854502477 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f621400f040 con 0x7f62200f6700 2021-11-19T08:45:10.580 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.589+0000 7f621d7fa700 1 -- 172.21.15.23:0/854502477 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6214014820 con 0x7f62200f6700 2021-11-19T08:45:10.580 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.589+0000 7f62259a2700 1 -- 172.21.15.23:0/854502477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62201078f0 con 0x7f62200f6700 2021-11-19T08:45:10.581 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.590+0000 7f62259a2700 1 -- 172.21.15.23:0/854502477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62200f4850 con 0x7f62200f6700 2021-11-19T08:45:10.581 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.590+0000 7f621d7fa700 1 -- 172.21.15.23:0/854502477 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f62140077b0 con 0x7f62200f6700 2021-11-19T08:45:10.582 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.591+0000 7f621d7fa700 1 --2- 172.21.15.23:0/854502477 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f620805c300 0x7f620805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:10.582 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.591+0000 7f621d7fa700 1 -- 172.21.15.23:0/854502477 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7f62140767a0 con 0x7f62200f6700 2021-11-19T08:45:10.582 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.591+0000 7f62249a0700 1 --2- 172.21.15.23:0/854502477 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f620805c300 0x7f620805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:10.582 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.591+0000 7f62249a0700 1 --2- 172.21.15.23:0/854502477 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f620805c300 0x7f620805e7c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f621800b620 tx=0x7f6218004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:10.583 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.595+0000 7f621d7fa700 1 -- 172.21.15.23:0/854502477 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6214048f00 con 0x7f62200f6700 2021-11-19T08:45:10.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:10 smithi023 conmon[33490]: cluster 2021-11-19T08:45:08.901326 2021-11-19T08:45:10.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:10 smithi023 conmon[33490]: +0000 mgr.smithi023.ehlpcl (mgr.14164) 334 : cluster [DBG] pgmap v311: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 341 B/s wr, 0 op/s 2021-11-19T08:45:10.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:10 smithi023 conmon[33490]: audit 2021-11-19T08:45:09.438536+0000 mon.smithi023 (mon.0) 907 : audit [DBG] from='client.? 172.21.15.23:0/3824574647' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:10.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:10 smithi023 conmon[33490]: 2021-11-19T08:45:10.773 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.789+0000 7f62259a2700 1 -- 172.21.15.23:0/854502477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6220045ad0 con 0x7f62200f6700 2021-11-19T08:45:10.774 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.789+0000 7f621d7fa700 1 -- 172.21.15.23:0/854502477 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2951 (secure 0 0 0) 0x7f621404c520 con 0x7f62200f6700 2021-11-19T08:45:10.774 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:10.775 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78},{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:58.255999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:10.776 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.791+0000 7f62259a2700 1 -- 172.21.15.23:0/854502477 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f620805c300 msgr2=0x7f620805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:10.776 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.791+0000 7f62259a2700 1 --2- 172.21.15.23:0/854502477 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f620805c300 0x7f620805e7c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f621800b620 tx=0x7f6218004d10).stop 2021-11-19T08:45:10.776 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.791+0000 7f62259a2700 1 -- 172.21.15.23:0/854502477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62200f6700 msgr2=0x7f6220106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:10.777 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.791+0000 7f62259a2700 1 --2- 172.21.15.23:0/854502477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62200f6700 0x7f6220106290 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f6214002700 tx=0x7f6214009e00).stop 2021-11-19T08:45:10.777 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.791+0000 7f62259a2700 1 -- 172.21.15.23:0/854502477 shutdown_connections 2021-11-19T08:45:10.777 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.791+0000 7f62259a2700 1 --2- 172.21.15.23:0/854502477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62200f5460 0x7f6220105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:10.777 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.791+0000 7f62259a2700 1 --2- 172.21.15.23:0/854502477 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f620805c300 0x7f620805e7c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:10.777 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.791+0000 7f62259a2700 1 --2- 172.21.15.23:0/854502477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62200f6700 0x7f6220106290 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:10.778 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.791+0000 7f62259a2700 1 -- 172.21.15.23:0/854502477 >> 172.21.15.23:0/854502477 conn(0x7f62200f0ba0 msgr2=0x7f62200fa150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:10.778 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.791+0000 7f62259a2700 1 -- 172.21.15.23:0/854502477 shutdown_connections 2021-11-19T08:45:10.778 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:10.791+0000 7f62259a2700 1 -- 172.21.15.23:0/854502477 wait complete. 2021-11-19T08:45:10.778 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 93 2021-11-19T08:45:10.788 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:58.255999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:10.788 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:rejoin 2021-11-19T08:45:11.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:11 smithi134 conmon[36363]: audit 2021-11-19T08:45:10.790318+0000 mon.smithi023 (mon.0) 908 : audit [DBG] from='client.? 172.21.15.23:0/854502477' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:11.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:11 smithi023 conmon[33490]: audit 2021-11-19T08:45:10 2021-11-19T08:45:11.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:11 smithi023 conmon[33490]: .790318+0000 mon.smithi023 (mon.0) 908 : audit [DBG] from='client.? 172.21.15.23:0/854502477' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:11.789 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:11.925 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.936+0000 7fe8a8655700 1 -- 172.21.15.23:0/627132374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a00f7920 msgr2=0x7fe8a00f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:11.925 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.936+0000 7fe8a8655700 1 --2- 172.21.15.23:0/627132374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a00f7920 0x7fe8a00f7d80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fe89c004660 tx=0x7fe89c009b30).stop 2021-11-19T08:45:11.925 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.937+0000 7fe8a8655700 1 -- 172.21.15.23:0/627132374 shutdown_connections 2021-11-19T08:45:11.926 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.937+0000 7fe8a8655700 1 --2- 172.21.15.23:0/627132374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a00f7920 0x7fe8a00f7d80 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:11.926 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.937+0000 7fe8a8655700 1 --2- 172.21.15.23:0/627132374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8a00f6680 0x7fe8a00f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:11.926 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.937+0000 7fe8a8655700 1 -- 172.21.15.23:0/627132374 >> 172.21.15.23:0/627132374 conn(0x7fe8a00f1de0 msgr2=0x7fe8a00f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:11.926 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.937+0000 7fe8a8655700 1 -- 172.21.15.23:0/627132374 shutdown_connections 2021-11-19T08:45:11.926 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.937+0000 7fe8a8655700 1 -- 172.21.15.23:0/627132374 wait complete. 2021-11-19T08:45:11.927 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.937+0000 7fe8a8655700 1 Processor -- start 2021-11-19T08:45:11.928 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.937+0000 7fe8a8655700 1 -- start start 2021-11-19T08:45:11.928 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.937+0000 7fe8a8655700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a00f6680 0x7fe8a0106e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:11.928 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.937+0000 7fe8a8655700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8a00f7920 0x7fe8a01072f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:11.928 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.937+0000 7fe8a8655700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a01078a0 con 0x7fe8a00f6680 2021-11-19T08:45:11.929 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.937+0000 7fe8a8655700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a01079e0 con 0x7fe8a00f7920 2021-11-19T08:45:11.929 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.937+0000 7fe8a63f1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a00f6680 0x7fe8a0106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:11.929 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.937+0000 7fe8a63f1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a00f6680 0x7fe8a0106e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48860/0 (socket says 172.21.15.23:48860) 2021-11-19T08:45:11.929 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.938+0000 7fe8a63f1700 1 -- 172.21.15.23:0/3682415164 learned_addr learned my addr 172.21.15.23:0/3682415164 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:11.930 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.938+0000 7fe8a5bf0700 1 --2- 172.21.15.23:0/3682415164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8a00f7920 0x7fe8a01072f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:11.930 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.938+0000 7fe8a63f1700 1 -- 172.21.15.23:0/3682415164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8a00f7920 msgr2=0x7fe8a01072f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:11.930 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.938+0000 7fe8a63f1700 1 --2- 172.21.15.23:0/3682415164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8a00f7920 0x7fe8a01072f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:11.931 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.938+0000 7fe8a63f1700 1 -- 172.21.15.23:0/3682415164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c005040 con 0x7fe8a00f6680 2021-11-19T08:45:11.931 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.938+0000 7fe8a63f1700 1 --2- 172.21.15.23:0/3682415164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a00f6680 0x7fe8a0106e00 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7fe89000e7c0 tx=0x7fe89000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:11.931 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.938+0000 7fe896ffd700 1 -- 172.21.15.23:0/3682415164 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8900092e0 con 0x7fe8a00f6680 2021-11-19T08:45:11.931 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.938+0000 7fe896ffd700 1 -- 172.21.15.23:0/3682415164 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe89000f040 con 0x7fe8a00f6680 2021-11-19T08:45:11.932 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.938+0000 7fe8a8655700 1 -- 172.21.15.23:0/3682415164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a010c480 con 0x7fe8a00f6680 2021-11-19T08:45:11.932 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.938+0000 7fe896ffd700 1 -- 172.21.15.23:0/3682415164 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8900147c0 con 0x7fe8a00f6680 2021-11-19T08:45:11.932 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.938+0000 7fe8a8655700 1 -- 172.21.15.23:0/3682415164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a010ca40 con 0x7fe8a00f6680 2021-11-19T08:45:11.932 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.939+0000 7fe8a8655700 1 -- 172.21.15.23:0/3682415164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a0027a00 con 0x7fe8a00f6680 2021-11-19T08:45:11.934 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.940+0000 7fe896ffd700 1 -- 172.21.15.23:0/3682415164 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe89001b030 con 0x7fe8a00f6680 2021-11-19T08:45:11.935 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.940+0000 7fe896ffd700 1 --2- 172.21.15.23:0/3682415164 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe88c05c2a0 0x7fe88c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:11.936 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.940+0000 7fe896ffd700 1 -- 172.21.15.23:0/3682415164 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fe890076320 con 0x7fe8a00f6680 2021-11-19T08:45:11.936 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.940+0000 7fe8a5bf0700 1 --2- 172.21.15.23:0/3682415164 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe88c05c2a0 0x7fe88c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:11.937 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.940+0000 7fe8a5bf0700 1 --2- 172.21.15.23:0/3682415164 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe88c05c2a0 0x7fe88c05e760 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fe89c005010 tx=0x7fe89c016040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:11.937 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:11.944+0000 7fe896ffd700 1 -- 172.21.15.23:0/3682415164 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe890048a80 con 0x7fe8a00f6680 2021-11-19T08:45:12.126 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:12.141+0000 7fe8a8655700 1 -- 172.21.15.23:0/3682415164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8a00fee10 con 0x7fe8a00f6680 2021-11-19T08:45:12.126 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:12.141+0000 7fe896ffd700 1 -- 172.21.15.23:0/3682415164 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2951 (secure 0 0 0) 0x7fe890019090 con 0x7fe8a00f6680 2021-11-19T08:45:12.127 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:12.127 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78},{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:58.255999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:12.128 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:12.143+0000 7fe8a8655700 1 -- 172.21.15.23:0/3682415164 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe88c05c2a0 msgr2=0x7fe88c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:12.128 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:12.143+0000 7fe8a8655700 1 --2- 172.21.15.23:0/3682415164 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe88c05c2a0 0x7fe88c05e760 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fe89c005010 tx=0x7fe89c016040).stop 2021-11-19T08:45:12.129 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:12.143+0000 7fe8a8655700 1 -- 172.21.15.23:0/3682415164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a00f6680 msgr2=0x7fe8a0106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:12.129 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:12.143+0000 7fe8a8655700 1 --2- 172.21.15.23:0/3682415164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a00f6680 0x7fe8a0106e00 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7fe89000e7c0 tx=0x7fe89000b040).stop 2021-11-19T08:45:12.129 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:12.143+0000 7fe8a8655700 1 -- 172.21.15.23:0/3682415164 shutdown_connections 2021-11-19T08:45:12.129 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:12.143+0000 7fe8a8655700 1 --2- 172.21.15.23:0/3682415164 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fe88c05c2a0 0x7fe88c05e760 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:12.130 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:12.143+0000 7fe8a8655700 1 --2- 172.21.15.23:0/3682415164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a00f6680 0x7fe8a0106e00 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:12.130 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:12.143+0000 7fe8a8655700 1 --2- 172.21.15.23:0/3682415164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8a00f7920 0x7fe8a01072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:12.130 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:12.143+0000 7fe8a8655700 1 -- 172.21.15.23:0/3682415164 >> 172.21.15.23:0/3682415164 conn(0x7fe8a00f1de0 msgr2=0x7fe8a00fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:12.130 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:12.143+0000 7fe8a8655700 1 -- 172.21.15.23:0/3682415164 shutdown_connections 2021-11-19T08:45:12.130 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:12.143+0000 7fe8a8655700 1 -- 172.21.15.23:0/3682415164 wait complete. 2021-11-19T08:45:12.131 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 93 2021-11-19T08:45:12.140 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:58.255999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:12.141 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:rejoin 2021-11-19T08:45:12.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:12 smithi023 conmon[33490]: cluster 2021-11-19T08:45:10.901953+0000 mgr.smithi023.ehlpcl 2021-11-19T08:45:12.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:12 smithi023 conmon[33490]: (mgr.14164) 335 : cluster [DBG] pgmap v312: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 341 B/s wr, 0 op/s 2021-11-19T08:45:12.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:12 smithi023 conmon[33490]: audit 2021-11-19T08:45 2021-11-19T08:45:12.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:12 smithi023 conmon[33490]: :12.142476+0000 mon.smithi023 (mon.0) 909 : audit 2021-11-19T08:45:12.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:12 smithi023 conmon[33490]: [DBG] from='client.? 172.21.15.23:0/3682415164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:12.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:12 smithi134 conmon[36363]: cluster 2021-11-19T08:45:10.901953+0000 mgr.smithi023.ehlpcl (mgr.14164) 335 : cluster [DBG] pgmap v312: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 341 B/s wr, 0 op/s 2021-11-19T08:45:12.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:12 smithi134 conmon[36363]: audit 2021-11-19T08:45:12.142476+0000 mon.smithi023 (mon.0) 909 : audit [DBG] from='client.? 172.21.15.23:0/3682415164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:13.142 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:13.279 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.290+0000 7fee035a7700 1 -- 172.21.15.23:0/2440595976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedfc0f6680 msgr2=0x7fedfc0f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:13.279 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.290+0000 7fee035a7700 1 --2- 172.21.15.23:0/2440595976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedfc0f6680 0x7fedfc0f6aa0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fedf40099e0 tx=0x7fedf4004d10).stop 2021-11-19T08:45:13.279 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.290+0000 7fee035a7700 1 -- 172.21.15.23:0/2440595976 shutdown_connections 2021-11-19T08:45:13.280 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.290+0000 7fee035a7700 1 --2- 172.21.15.23:0/2440595976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fedfc0f7920 0x7fedfc0f7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:13.280 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.290+0000 7fee035a7700 1 --2- 172.21.15.23:0/2440595976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedfc0f6680 0x7fedfc0f6aa0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:13.280 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.290+0000 7fee035a7700 1 -- 172.21.15.23:0/2440595976 >> 172.21.15.23:0/2440595976 conn(0x7fedfc0f1de0 msgr2=0x7fedfc0f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:13.281 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.290+0000 7fee035a7700 1 -- 172.21.15.23:0/2440595976 shutdown_connections 2021-11-19T08:45:13.281 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.290+0000 7fee035a7700 1 -- 172.21.15.23:0/2440595976 wait complete. 2021-11-19T08:45:13.281 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.291+0000 7fee035a7700 1 Processor -- start 2021-11-19T08:45:13.281 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.291+0000 7fee035a7700 1 -- start start 2021-11-19T08:45:13.281 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.291+0000 7fee035a7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fedfc0f6680 0x7fedfc06b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:13.282 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.291+0000 7fee035a7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedfc0f7920 0x7fedfc069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:13.282 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.291+0000 7fee035a7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc06bd70 con 0x7fedfc0f7920 2021-11-19T08:45:13.282 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.291+0000 7fee035a7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc10ac90 con 0x7fedfc0f6680 2021-11-19T08:45:13.282 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.291+0000 7fee00b42700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedfc0f7920 0x7fedfc069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:13.283 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.291+0000 7fee00b42700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedfc0f7920 0x7fedfc069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48872/0 (socket says 172.21.15.23:48872) 2021-11-19T08:45:13.283 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.291+0000 7fee00b42700 1 -- 172.21.15.23:0/2608993078 learned_addr learned my addr 172.21.15.23:0/2608993078 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:13.283 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.291+0000 7fee01343700 1 --2- 172.21.15.23:0/2608993078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fedfc0f6680 0x7fedfc06b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:13.283 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.291+0000 7fee00b42700 1 -- 172.21.15.23:0/2608993078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fedfc0f6680 msgr2=0x7fedfc06b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:13.284 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.292+0000 7fee00b42700 1 --2- 172.21.15.23:0/2608993078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fedfc0f6680 0x7fedfc06b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:13.284 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.292+0000 7fee00b42700 1 -- 172.21.15.23:0/2608993078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf4009640 con 0x7fedfc0f7920 2021-11-19T08:45:13.284 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.292+0000 7fee00b42700 1 --2- 172.21.15.23:0/2608993078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedfc0f7920 0x7fedfc069e20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7fedf000c790 tx=0x7fedf0007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:13.284 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.292+0000 7fedf9ffb700 1 -- 172.21.15.23:0/2608993078 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedf000e680 con 0x7fedfc0f7920 2021-11-19T08:45:13.284 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.292+0000 7fee035a7700 1 -- 172.21.15.23:0/2608993078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedfc06a310 con 0x7fedfc0f7920 2021-11-19T08:45:13.285 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.292+0000 7fedf9ffb700 1 -- 172.21.15.23:0/2608993078 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedf000ec90 con 0x7fedfc0f7920 2021-11-19T08:45:13.285 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.292+0000 7fedf9ffb700 1 -- 172.21.15.23:0/2608993078 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedf0013d60 con 0x7fedfc0f7920 2021-11-19T08:45:13.285 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.292+0000 7fee035a7700 1 -- 172.21.15.23:0/2608993078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedfc06a850 con 0x7fedfc0f7920 2021-11-19T08:45:13.285 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.293+0000 7fee035a7700 1 -- 172.21.15.23:0/2608993078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedfc10ae50 con 0x7fedfc0f7920 2021-11-19T08:45:13.286 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.294+0000 7fedf9ffb700 1 -- 172.21.15.23:0/2608993078 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fedf001a060 con 0x7fedfc0f7920 2021-11-19T08:45:13.287 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.294+0000 7fedf9ffb700 1 --2- 172.21.15.23:0/2608993078 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fede405c2a0 0x7fede405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:13.287 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.294+0000 7fedf9ffb700 1 -- 172.21.15.23:0/2608993078 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6758+0+0 (secure 0 0 0) 0x7fedf0076670 con 0x7fedfc0f7920 2021-11-19T08:45:13.287 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.294+0000 7fee01343700 1 --2- 172.21.15.23:0/2608993078 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fede405c2a0 0x7fede405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:13.287 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.294+0000 7fee01343700 1 --2- 172.21.15.23:0/2608993078 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fede405c2a0 0x7fede405e760 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fedf400b3b0 tx=0x7fedf4004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:13.288 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.298+0000 7fedf9ffb700 1 -- 172.21.15.23:0/2608993078 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fedf004c7a0 con 0x7fedfc0f7920 2021-11-19T08:45:13.478 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.493+0000 7fee035a7700 1 -- 172.21.15.23:0/2608993078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedfc06a4a0 con 0x7fedfc0f7920 2021-11-19T08:45:13.479 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.493+0000 7fedf9ffb700 1 -- 172.21.15.23:0/2608993078 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+2951 (secure 0 0 0) 0x7fedfc06a4a0 con 0x7fedfc0f7920 2021-11-19T08:45:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:13.480 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":78},{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":92,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:44:58.255999+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14880,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14880":{"gid":14880,"name":"cephfs.smithi023.fuvzkr","rank":0,"incarnation":85,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.23:6827/3968768531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3968768531},{"type":"v1","addr":"172.21.15.23:6827","nonce":3968768531}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:13.481 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.495+0000 7fee035a7700 1 -- 172.21.15.23:0/2608993078 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fede405c2a0 msgr2=0x7fede405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:13.482 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.495+0000 7fee035a7700 1 --2- 172.21.15.23:0/2608993078 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fede405c2a0 0x7fede405e760 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fedf400b3b0 tx=0x7fedf4004d10).stop 2021-11-19T08:45:13.482 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.495+0000 7fee035a7700 1 -- 172.21.15.23:0/2608993078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedfc0f7920 msgr2=0x7fedfc069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:13.482 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.495+0000 7fee035a7700 1 --2- 172.21.15.23:0/2608993078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedfc0f7920 0x7fedfc069e20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7fedf000c790 tx=0x7fedf0007d70).stop 2021-11-19T08:45:13.482 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.495+0000 7fee035a7700 1 -- 172.21.15.23:0/2608993078 shutdown_connections 2021-11-19T08:45:13.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.495+0000 7fee035a7700 1 --2- 172.21.15.23:0/2608993078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fedfc0f6680 0x7fedfc06b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:13.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.495+0000 7fee035a7700 1 --2- 172.21.15.23:0/2608993078 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fede405c2a0 0x7fede405e760 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:13.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.495+0000 7fee035a7700 1 --2- 172.21.15.23:0/2608993078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedfc0f7920 0x7fedfc069e20 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:13.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.495+0000 7fee035a7700 1 -- 172.21.15.23:0/2608993078 >> 172.21.15.23:0/2608993078 conn(0x7fedfc0f1de0 msgr2=0x7fedfc0fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:13.483 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.495+0000 7fee035a7700 1 -- 172.21.15.23:0/2608993078 shutdown_connections 2021-11-19T08:45:13.484 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:13.495+0000 7fee035a7700 1 -- 172.21.15.23:0/2608993078 wait complete. 2021-11-19T08:45:13.484 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 93 2021-11-19T08:45:13.492 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:44:58.255999+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 14880, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14880': {'gid': 14880, 'name': 'cephfs.smithi023.fuvzkr', 'rank': 0, 'incarnation': 85, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.23:6827/3968768531', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6826', 'nonce': 3968768531}, {'type': 'v1', 'addr': '172.21.15.23:6827', 'nonce': 3968768531}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:13.493 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_14880:up:rejoin 2021-11-19T08:45:14.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:14.520 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: cluster 2021-11-19T08:45:12.902465 2021-11-19T08:45:14.520 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: +0000 mgr.smithi023.ehlpcl (mgr.14164) 336 : cluster [DBG] pgmap v313: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 85 B/s wr, 0 op/s 2021-11-19T08:45:14.521 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: audit 2021-11-19T08: 2021-11-19T08:45:14.521 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: 45:13.495409+0000 mon.smithi023 (mon.0) 910 : audit [DBG] 2021-11-19T08:45:14.521 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/2608993078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:14.522 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: cluster 2021-11-19 2021-11-19T08:45:14.522 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: T08:45:14.240156+0000 2021-11-19T08:45:14.523 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: mon.smithi023 (mon.0) 911 : cluster [WRN] 2021-11-19T08:45:14.523 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: Replacing daemon mds.cephfs.smithi023.fuvzkr as rank 0 with standby daemon mds.cephfs.smithi134.jqsnzo 2021-11-19T08:45:14.523 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: cluster 2021-11- 2021-11-19T08:45:14.524 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: 19T08:45:14.240223+0000 2021-11-19T08:45:14.524 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: mon.smithi023 (mon.0) 912 : cluster [INF] 2021-11-19T08:45:14.524 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: MDS daemon mds.cephfs.smithi023.fuvzkr is removed because it is dead or otherwise unavailable. 2021-11-19T08:45:14.525 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: cluster 2021-11- 2021-11-19T08:45:14.525 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: 19T08:45:14.249723 2021-11-19T08:45:14.525 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: +0000 mon.smithi023 (mon.0) 913 : cluster 2021-11-19T08:45:14.525 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:14 smithi023 conmon[33490]: [DBG] osdmap e58: 6 total, 6 up, 6 in 2021-11-19T08:45:14.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:14 smithi134 conmon[36363]: cluster 2021-11-19T08:45:12.902465+0000 mgr.smithi023.ehlpcl (mgr.14164) 336 : cluster [DBG] pgmap v313: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 85 B/s wr, 0 op/s 2021-11-19T08:45:14.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:14 smithi134 conmon[36363]: 2021-11-19T08:45:14.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:14 smithi134 conmon[36363]: audit 2021-11-19T08:45:13.495409+0000 mon.smithi023 (mon.0) 910 : audit [DBG] from='client.? 172.21.15.23:0/2608993078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:14.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:14 smithi134 conmon[36363]: cluster 2021-11-19T08:45:14.240156+0000 mon.smithi023 (mon.0) 911 : 2021-11-19T08:45:14.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:14 smithi134 conmon[36363]: cluster [WRN] Replacing daemon mds.cephfs.smithi023.fuvzkr as rank 0 with standby daemon mds.cephfs.smithi134.jqsnzo 2021-11-19T08:45:14.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:14 smithi134 conmon[36363]: cluster 2021 2021-11-19T08:45:14.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:14 smithi134 conmon[36363]: -11-19T08:45:14.240223+0000 mon.smithi023 (mon.0) 912 : cluster [INF] MDS daemon mds.cephfs.smithi023.fuvzkr is removed because it is dead or otherwise unavailable. 2021-11-19T08:45:14.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:14 smithi134 conmon[36363]: 2021-11-19T08:45:14.545 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:14 smithi134 conmon[36363]: cluster 2021-11-19T08:45:14.545 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:14 smithi134 conmon[36363]: 2021-11-19T08:45:14.249723+0000 mon.smithi023 (mon.0) 913 2021-11-19T08:45:14.545 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:14 smithi134 conmon[36363]: : cluster [DBG] osdmap e58: 6 total, 6 up, 6 in 2021-11-19T08:45:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.636+0000 7f0efee74700 1 -- 172.21.15.23:0/1279657663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef80f55a0 msgr2=0x7f0ef80f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.636+0000 7f0efee74700 1 --2- 172.21.15.23:0/1279657663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef80f55a0 0x7f0ef80f59c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f0ef0004660 tx=0x7f0ef0009b30).stop 2021-11-19T08:45:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.636+0000 7f0efee74700 1 -- 172.21.15.23:0/1279657663 shutdown_connections 2021-11-19T08:45:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.636+0000 7f0efee74700 1 --2- 172.21.15.23:0/1279657663 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ef80f6840 0x7f0ef80f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.636+0000 7f0efee74700 1 --2- 172.21.15.23:0/1279657663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef80f55a0 0x7f0ef80f59c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.636+0000 7f0efee74700 1 -- 172.21.15.23:0/1279657663 >> 172.21.15.23:0/1279657663 conn(0x7f0ef80f0ce0 msgr2=0x7f0ef80f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.636+0000 7f0efee74700 1 -- 172.21.15.23:0/1279657663 shutdown_connections 2021-11-19T08:45:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.636+0000 7f0efee74700 1 -- 172.21.15.23:0/1279657663 wait complete. 2021-11-19T08:45:14.628 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.636+0000 7f0efee74700 1 Processor -- start 2021-11-19T08:45:14.628 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.636+0000 7f0efee74700 1 -- start start 2021-11-19T08:45:14.628 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0efee74700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ef80f55a0 0x7f0ef81090d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:14.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0efee74700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef80f6840 0x7f0ef81095c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:14.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0efee74700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef8109b70 con 0x7f0ef80f6840 2021-11-19T08:45:14.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0efee74700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef8109cb0 con 0x7f0ef80f55a0 2021-11-19T08:45:14.629 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0ef7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef80f6840 0x7f0ef81095c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:14.630 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0ef7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef80f6840 0x7f0ef81095c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48884/0 (socket says 172.21.15.23:48884) 2021-11-19T08:45:14.630 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0ef7fff700 1 -- 172.21.15.23:0/2613114630 learned_addr learned my addr 172.21.15.23:0/2613114630 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:14.630 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0efcc10700 1 --2- 172.21.15.23:0/2613114630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ef80f55a0 0x7f0ef81090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:14.631 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0ef7fff700 1 -- 172.21.15.23:0/2613114630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ef80f55a0 msgr2=0x7f0ef81090d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:14.631 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0ef7fff700 1 --2- 172.21.15.23:0/2613114630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ef80f55a0 0x7f0ef81090d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:14.631 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0ef7fff700 1 -- 172.21.15.23:0/2613114630 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef0005040 con 0x7f0ef80f6840 2021-11-19T08:45:14.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0ef7fff700 1 --2- 172.21.15.23:0/2613114630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef80f6840 0x7f0ef81095c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f0ee8007fd0 tx=0x7f0ee8004ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:14.632 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0ef57fa700 1 -- 172.21.15.23:0/2613114630 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ee800b4f0 con 0x7f0ef80f6840 2021-11-19T08:45:14.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0ef57fa700 1 -- 172.21.15.23:0/2613114630 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ee800f040 con 0x7f0ef80f6840 2021-11-19T08:45:14.633 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0efee74700 1 -- 172.21.15.23:0/2613114630 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef810e750 con 0x7f0ef80f6840 2021-11-19T08:45:14.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.637+0000 7f0ef57fa700 1 -- 172.21.15.23:0/2613114630 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ee8013a00 con 0x7f0ef80f6840 2021-11-19T08:45:14.634 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.638+0000 7f0efee74700 1 -- 172.21.15.23:0/2613114630 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef810ed10 con 0x7f0ef80f6840 2021-11-19T08:45:14.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.638+0000 7f0efee74700 1 -- 172.21.15.23:0/2613114630 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef80fdd30 con 0x7f0ef80f6840 2021-11-19T08:45:14.635 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.640+0000 7f0ef57fa700 1 -- 172.21.15.23:0/2613114630 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f0ee801a030 con 0x7f0ef80f6840 2021-11-19T08:45:14.636 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.640+0000 7f0ef57fa700 1 --2- 172.21.15.23:0/2613114630 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0ee405c2a0 0x7f0ee405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:14.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.640+0000 7f0ef57fa700 1 -- 172.21.15.23:0/2613114630 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f0ee8076a60 con 0x7f0ef80f6840 2021-11-19T08:45:14.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.640+0000 7f0efcc10700 1 --2- 172.21.15.23:0/2613114630 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0ee405c2a0 0x7f0ee405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:14.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.640+0000 7f0efcc10700 1 --2- 172.21.15.23:0/2613114630 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0ee405c2a0 0x7f0ee405e760 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f0ef0005010 tx=0x7f0ef0009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:14.637 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.643+0000 7f0ef57fa700 1 -- 172.21.15.23:0/2613114630 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ee80492a0 con 0x7f0ef80f6840 2021-11-19T08:45:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.843+0000 7f0efee74700 1 -- 172.21.15.23:0/2613114630 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ef810f410 con 0x7f0ef80f6840 2021-11-19T08:45:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.843+0000 7f0ef57fa700 1 -- 172.21.15.23:0/2613114630 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+2575 (secure 0 0 0) 0x7f0ee8018020 con 0x7f0ef80f6840 2021-11-19T08:45:14.829 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:14.830 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":94,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:45:14.241044+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24565,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24565":{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":94,"state":"up:replay","state_seq":3,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.845+0000 7f0efee74700 1 -- 172.21.15.23:0/2613114630 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0ee405c2a0 msgr2=0x7f0ee405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.845+0000 7f0efee74700 1 --2- 172.21.15.23:0/2613114630 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0ee405c2a0 0x7f0ee405e760 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f0ef0005010 tx=0x7f0ef0009b30).stop 2021-11-19T08:45:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.845+0000 7f0efee74700 1 -- 172.21.15.23:0/2613114630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef80f6840 msgr2=0x7f0ef81095c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.845+0000 7f0efee74700 1 --2- 172.21.15.23:0/2613114630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef80f6840 0x7f0ef81095c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f0ee8007fd0 tx=0x7f0ee8004ee0).stop 2021-11-19T08:45:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.845+0000 7f0efee74700 1 -- 172.21.15.23:0/2613114630 shutdown_connections 2021-11-19T08:45:14.832 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.845+0000 7f0efee74700 1 --2- 172.21.15.23:0/2613114630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ef80f55a0 0x7f0ef81090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.845+0000 7f0efee74700 1 --2- 172.21.15.23:0/2613114630 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f0ee405c2a0 0x7f0ee405e760 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.845+0000 7f0efee74700 1 --2- 172.21.15.23:0/2613114630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef80f6840 0x7f0ef81095c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.845+0000 7f0efee74700 1 -- 172.21.15.23:0/2613114630 >> 172.21.15.23:0/2613114630 conn(0x7f0ef80f0ce0 msgr2=0x7f0ef805fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.845+0000 7f0efee74700 1 -- 172.21.15.23:0/2613114630 shutdown_connections 2021-11-19T08:45:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:14.845+0000 7f0efee74700 1 -- 172.21.15.23:0/2613114630 wait complete. 2021-11-19T08:45:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 94 2021-11-19T08:45:14.843 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 94, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:45:14.241044+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24565, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 94, 'state': 'up:replay', 'state_seq': 3, 'addr': '172.21.15.134:6825/955716436', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 955716436}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 955716436}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:14.843 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:replay 2021-11-19T08:45:15.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:15 smithi134 conmon[36363]: cluster 2021-11-19T08:45:14.254953+0000 mon.smithi023 (mon.0) 914 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:replay,1=cephfs.smithi023.wazanl=up:active} 1 up:standby 2021-11-19T08:45:15.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:15 smithi134 conmon[36363]: audit 2021-11-19T08:45:14.845335+0000 mon.smithi023 (mon.0) 915 : audit [DBG] from='client.? 172.21.15.23:0/2613114630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:15.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:15 smithi023 conmon[33490]: cluster 2021-11-19T08:45:14.254953+0000 mon.smithi023 (mon.0) 914 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:replay,1=cephfs.smithi023.wazanl=up:active} 1 up:standby 2021-11-19T08:45:15.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:15 smithi023 conmon[33490]: audit 2021-11-19T08:45:14.845335+0000 mon.smithi023 (mon.0) 915 : audit [DBG] from='client.? 172.21.15.23:0/2613114630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:15.844 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:15.983 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.992+0000 7f06f346b700 1 -- 172.21.15.23:0/1566906450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ec0f6820 msgr2=0x7f06ec0f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:15.984 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.992+0000 7f06f346b700 1 --2- 172.21.15.23:0/1566906450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ec0f6820 0x7f06ec0f6c80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f06e0004660 tx=0x7f06e0009b30).stop 2021-11-19T08:45:15.984 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.992+0000 7f06f346b700 1 -- 172.21.15.23:0/1566906450 shutdown_connections 2021-11-19T08:45:15.985 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.992+0000 7f06f346b700 1 --2- 172.21.15.23:0/1566906450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ec0f6820 0x7f06ec0f6c80 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:15.985 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.992+0000 7f06f346b700 1 --2- 172.21.15.23:0/1566906450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06ec0f5580 0x7f06ec0f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:15.986 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.992+0000 7f06f346b700 1 -- 172.21.15.23:0/1566906450 >> 172.21.15.23:0/1566906450 conn(0x7f06ec0f0ce0 msgr2=0x7f06ec0f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:15.986 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.992+0000 7f06f346b700 1 -- 172.21.15.23:0/1566906450 shutdown_connections 2021-11-19T08:45:15.987 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.992+0000 7f06f346b700 1 -- 172.21.15.23:0/1566906450 wait complete. 2021-11-19T08:45:15.987 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.992+0000 7f06f346b700 1 Processor -- start 2021-11-19T08:45:15.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.992+0000 7f06f346b700 1 -- start start 2021-11-19T08:45:15.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.992+0000 7f06f346b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06ec0f5580 0x7f06ec105dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:15.988 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06f346b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ec0f6820 0x7f06ec1062b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:15.989 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06f346b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec106860 con 0x7f06ec0f6820 2021-11-19T08:45:15.989 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06f346b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec1069a0 con 0x7f06ec0f5580 2021-11-19T08:45:15.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06f0a06700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ec0f6820 0x7f06ec1062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:15.990 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06f0a06700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ec0f6820 0x7f06ec1062b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48896/0 (socket says 172.21.15.23:48896) 2021-11-19T08:45:15.991 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06f0a06700 1 -- 172.21.15.23:0/1481907994 learned_addr learned my addr 172.21.15.23:0/1481907994 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:15.991 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06f1207700 1 --2- 172.21.15.23:0/1481907994 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06ec0f5580 0x7f06ec105dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:15.992 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06f0a06700 1 -- 172.21.15.23:0/1481907994 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06ec0f5580 msgr2=0x7f06ec105dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:15.992 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06f0a06700 1 --2- 172.21.15.23:0/1481907994 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06ec0f5580 0x7f06ec105dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:15.992 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06f0a06700 1 -- 172.21.15.23:0/1481907994 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e0005040 con 0x7f06ec0f6820 2021-11-19T08:45:15.993 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06f0a06700 1 --2- 172.21.15.23:0/1481907994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ec0f6820 0x7f06ec1062b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f06e0009830 tx=0x7f06e0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:15.993 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06e9ffb700 1 -- 172.21.15.23:0/1481907994 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06e0018070 con 0x7f06ec0f6820 2021-11-19T08:45:15.994 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06e9ffb700 1 -- 172.21.15.23:0/1481907994 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06e0013410 con 0x7f06ec0f6820 2021-11-19T08:45:15.994 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06f346b700 1 -- 172.21.15.23:0/1481907994 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec10b3e0 con 0x7f06ec0f6820 2021-11-19T08:45:15.995 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06e9ffb700 1 -- 172.21.15.23:0/1481907994 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06e001c620 con 0x7f06ec0f6820 2021-11-19T08:45:15.995 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.993+0000 7f06f346b700 1 -- 172.21.15.23:0/1481907994 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec10b9a0 con 0x7f06ec0f6820 2021-11-19T08:45:15.995 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.994+0000 7f06f346b700 1 -- 172.21.15.23:0/1481907994 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec027a00 con 0x7f06ec0f6820 2021-11-19T08:45:15.996 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.995+0000 7f06e9ffb700 1 -- 172.21.15.23:0/1481907994 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f06e001c8e0 con 0x7f06ec0f6820 2021-11-19T08:45:15.996 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.995+0000 7f06e9ffb700 1 --2- 172.21.15.23:0/1481907994 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f06d405c2a0 0x7f06d405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:15.997 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.995+0000 7f06e9ffb700 1 -- 172.21.15.23:0/1481907994 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f06e0078350 con 0x7f06ec0f6820 2021-11-19T08:45:15.997 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.995+0000 7f06f1207700 1 --2- 172.21.15.23:0/1481907994 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f06d405c2a0 0x7f06d405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:15.998 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.996+0000 7f06f1207700 1 --2- 172.21.15.23:0/1481907994 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f06d405c2a0 0x7f06d405e760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f06ec0f6680 tx=0x7f06dc009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:15.999 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:15.999+0000 7f06e9ffb700 1 -- 172.21.15.23:0/1481907994 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06e004aa00 con 0x7f06ec0f6820 2021-11-19T08:45:16.183 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:16.198+0000 7f06f346b700 1 -- 172.21.15.23:0/1481907994 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06ec0fdd10 con 0x7f06ec0f6820 2021-11-19T08:45:16.184 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:16.198+0000 7f06e9ffb700 1 -- 172.21.15.23:0/1481907994 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2577 (secure 0 0 0) 0x7f06e004e020 con 0x7f06ec0f6820 2021-11-19T08:45:16.184 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:16.185 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":95,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:45:15.258264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24565,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24565":{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":94,"state":"up:resolve","state_seq":19,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:16.186 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:16.200+0000 7f06f346b700 1 -- 172.21.15.23:0/1481907994 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f06d405c2a0 msgr2=0x7f06d405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:16.186 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:16.200+0000 7f06f346b700 1 --2- 172.21.15.23:0/1481907994 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f06d405c2a0 0x7f06d405e760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f06ec0f6680 tx=0x7f06dc009040).stop 2021-11-19T08:45:16.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:16.200+0000 7f06f346b700 1 -- 172.21.15.23:0/1481907994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ec0f6820 msgr2=0x7f06ec1062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:16.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:16.200+0000 7f06f346b700 1 --2- 172.21.15.23:0/1481907994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ec0f6820 0x7f06ec1062b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7f06e0009830 tx=0x7f06e0009b30).stop 2021-11-19T08:45:16.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:16.200+0000 7f06f346b700 1 -- 172.21.15.23:0/1481907994 shutdown_connections 2021-11-19T08:45:16.187 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:16.200+0000 7f06f346b700 1 --2- 172.21.15.23:0/1481907994 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06ec0f5580 0x7f06ec105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:16.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:16.200+0000 7f06f346b700 1 --2- 172.21.15.23:0/1481907994 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f06d405c2a0 0x7f06d405e760 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:16.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:16.200+0000 7f06f346b700 1 --2- 172.21.15.23:0/1481907994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06ec0f6820 0x7f06ec1062b0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:16.188 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:16.200+0000 7f06f346b700 1 -- 172.21.15.23:0/1481907994 >> 172.21.15.23:0/1481907994 conn(0x7f06ec0f0ce0 msgr2=0x7f06ec0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:16.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:16.200+0000 7f06f346b700 1 -- 172.21.15.23:0/1481907994 shutdown_connections 2021-11-19T08:45:16.189 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:16.200+0000 7f06f346b700 1 -- 172.21.15.23:0/1481907994 wait complete. 2021-11-19T08:45:16.189 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 95 2021-11-19T08:45:16.197 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:45:15.258264+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24565, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 94, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.134:6825/955716436', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 955716436}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 955716436}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:16.198 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:resolve 2021-11-19T08:45:16.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:16 smithi134 conmon[36363]: cluster 2021-11-19T08:45:14.902927+0000 mgr.smithi023.ehlpcl (mgr.14164) 337 : cluster [DBG] pgmap v315: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-19T08:45:16.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:16 smithi134 conmon[36363]: cluster 2021-11-19 2021-11-19T08:45:16.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:16 smithi134 conmon[36363]: T08:45:15.266822+0000 mon.smithi023 (mon.0) 916 : cluster [DBG] mds.? [v2:172.21.15.134:6824/955716436,v1:172.21.15.134:6825/955716436] up:resolve 2021-11-19T08:45:16.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:16 smithi134 conmon[36363]: cluster 2021-11-19T08:45:15.266895+0000 mon.smithi023 (mon.0) 917 : cluster 2021-11-19T08:45:16.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:16 smithi134 conmon[36363]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.wazanl=up:active} 1 up:standby 2021-11-19T08:45:16.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:16 smithi134 conmon[36363]: audit 2021-11-19T08:45:16.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:16 smithi134 conmon[36363]: 2021-11-19T08:45:16.200273 2021-11-19T08:45:16.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:16 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 918 : audit [DBG] 2021-11-19T08:45:16.542 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:16 smithi134 conmon[36363]: from='client.? 172.21.15.23:0/1481907994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:16.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:16 smithi023 conmon[33490]: cluster 2021-11-19T08:45:14.902927+0000 mgr.smithi023.ehlpcl 2021-11-19T08:45:16.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:16 smithi023 conmon[33490]: (mgr.14164) 337 : cluster [DBG] pgmap v315: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2021-11-19T08:45:16.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:16 smithi023 conmon[33490]: cluster 2021-11-19T08:45:15.266822+0000 mon.smithi023 (mon. 2021-11-19T08:45:16.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:16 smithi023 conmon[33490]: 0) 916 : cluster [DBG] mds.? [v2:172.21.15.134:6824/955716436,v1:172.21.15.134:6825/955716436] up:resolve 2021-11-19T08:45:16.643 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:16 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:45:16.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:16 smithi023 conmon[33490]: 08:45:15.266895+0000 mon.smithi023 (mon.0) 917 : cluster 2021-11-19T08:45:16.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:16 smithi023 conmon[33490]: [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi023.wazanl=up:active} 1 up:standby 2021-11-19T08:45:16.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:16 smithi023 conmon[33490]: audit 2021 2021-11-19T08:45:16.644 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:16 smithi023 conmon[33490]: -11-19T08:45:16.200273+0000 mon.smithi023 ( 2021-11-19T08:45:16.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:16 smithi023 conmon[33490]: mon.0) 918 : audit [DBG] from='client.? 172.21.15.23:0/1481907994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:16.645 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:16 smithi023 conmon[33490]: 2021-11-19T08:45:17.199 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:17.332 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.346+0000 7f5615832700 1 -- 172.21.15.23:0/806519137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56100f55a0 msgr2=0x7f56100f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:17.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.346+0000 7f5615832700 1 --2- 172.21.15.23:0/806519137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56100f55a0 0x7f56100f59c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f56080099e0 tx=0x7f5608004d10).stop 2021-11-19T08:45:17.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.346+0000 7f5615832700 1 -- 172.21.15.23:0/806519137 shutdown_connections 2021-11-19T08:45:17.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.346+0000 7f5615832700 1 --2- 172.21.15.23:0/806519137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56100f6840 0x7f56100f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:17.333 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.346+0000 7f5615832700 1 --2- 172.21.15.23:0/806519137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56100f55a0 0x7f56100f59c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:17.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.346+0000 7f5615832700 1 -- 172.21.15.23:0/806519137 >> 172.21.15.23:0/806519137 conn(0x7f56100f0ce0 msgr2=0x7f56100f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:17.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.346+0000 7f5615832700 1 -- 172.21.15.23:0/806519137 shutdown_connections 2021-11-19T08:45:17.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.346+0000 7f5615832700 1 -- 172.21.15.23:0/806519137 wait complete. 2021-11-19T08:45:17.334 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.347+0000 7f5615832700 1 Processor -- start 2021-11-19T08:45:17.335 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.347+0000 7f5615832700 1 -- start start 2021-11-19T08:45:17.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.347+0000 7f5615832700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56100f55a0 0x7f56101091c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:17.337 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.347+0000 7f5615832700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56100f6840 0x7f56101096b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:17.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.347+0000 7f5615832700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5610109c60 con 0x7f56100f6840 2021-11-19T08:45:17.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.347+0000 7f5615832700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5610109da0 con 0x7f56100f55a0 2021-11-19T08:45:17.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.347+0000 7f560e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56100f6840 0x7f56101096b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:17.338 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.347+0000 7f560e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56100f6840 0x7f56101096b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48908/0 (socket says 172.21.15.23:48908) 2021-11-19T08:45:17.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.348+0000 7f560e7fc700 1 -- 172.21.15.23:0/3894014757 learned_addr learned my addr 172.21.15.23:0/3894014757 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:17.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.348+0000 7f560effd700 1 --2- 172.21.15.23:0/3894014757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56100f55a0 0x7f56101091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:17.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.348+0000 7f560e7fc700 1 -- 172.21.15.23:0/3894014757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56100f55a0 msgr2=0x7f56101091c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:17.339 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.348+0000 7f560e7fc700 1 --2- 172.21.15.23:0/3894014757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56100f55a0 0x7f56101091c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:17.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.348+0000 7f560e7fc700 1 -- 172.21.15.23:0/3894014757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5608009640 con 0x7f56100f6840 2021-11-19T08:45:17.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.348+0000 7f560e7fc700 1 --2- 172.21.15.23:0/3894014757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56100f6840 0x7f56101096b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f560400c530 tx=0x7f5604007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:17.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.348+0000 7f55f77fe700 1 -- 172.21.15.23:0/3894014757 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f560400f650 con 0x7f56100f6840 2021-11-19T08:45:17.340 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.348+0000 7f55f77fe700 1 -- 172.21.15.23:0/3894014757 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f560400fc60 con 0x7f56100f6840 2021-11-19T08:45:17.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.348+0000 7f5615832700 1 -- 172.21.15.23:0/3894014757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f561010e840 con 0x7f56100f6840 2021-11-19T08:45:17.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.348+0000 7f55f77fe700 1 -- 172.21.15.23:0/3894014757 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5604012d30 con 0x7f56100f6840 2021-11-19T08:45:17.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.348+0000 7f5615832700 1 -- 172.21.15.23:0/3894014757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561010ed80 con 0x7f56100f6840 2021-11-19T08:45:17.341 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.349+0000 7f5615832700 1 -- 172.21.15.23:0/3894014757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56100fdd30 con 0x7f56100f6840 2021-11-19T08:45:17.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.350+0000 7f55f77fe700 1 -- 172.21.15.23:0/3894014757 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f5604010070 con 0x7f56100f6840 2021-11-19T08:45:17.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.350+0000 7f55f77fe700 1 --2- 172.21.15.23:0/3894014757 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f55f805c2a0 0x7f55f805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:17.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.350+0000 7f55f77fe700 1 -- 172.21.15.23:0/3894014757 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f5604076200 con 0x7f56100f6840 2021-11-19T08:45:17.342 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.350+0000 7f560effd700 1 --2- 172.21.15.23:0/3894014757 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f55f805c2a0 0x7f55f805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:17.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.350+0000 7f560effd700 1 --2- 172.21.15.23:0/3894014757 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f55f805c2a0 0x7f55f805e760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f560800b3b0 tx=0x7f5608004d10).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:17.343 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.354+0000 7f55f77fe700 1 -- 172.21.15.23:0/3894014757 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56040452b0 con 0x7f56100f6840 2021-11-19T08:45:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.549+0000 7f5615832700 1 -- 172.21.15.23:0/3894014757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f561010f4f0 con 0x7f56100f6840 2021-11-19T08:45:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.550+0000 7f55f77fe700 1 -- 172.21.15.23:0/3894014757 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2577 (secure 0 0 0) 0x7f5604048910 con 0x7f56100f6840 2021-11-19T08:45:17.536 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:17.536 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":95,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:45:15.258264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24565,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24565":{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":94,"state":"up:resolve","state_seq":19,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:17.538 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.551+0000 7f5615832700 1 -- 172.21.15.23:0/3894014757 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f55f805c2a0 msgr2=0x7f55f805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:17.538 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.551+0000 7f5615832700 1 --2- 172.21.15.23:0/3894014757 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f55f805c2a0 0x7f55f805e760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f560800b3b0 tx=0x7f5608004d10).stop 2021-11-19T08:45:17.538 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.551+0000 7f5615832700 1 -- 172.21.15.23:0/3894014757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56100f6840 msgr2=0x7f56101096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:17.538 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.551+0000 7f5615832700 1 --2- 172.21.15.23:0/3894014757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56100f6840 0x7f56101096b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f560400c530 tx=0x7f5604007e20).stop 2021-11-19T08:45:17.539 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.551+0000 7f5615832700 1 -- 172.21.15.23:0/3894014757 shutdown_connections 2021-11-19T08:45:17.539 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.551+0000 7f5615832700 1 --2- 172.21.15.23:0/3894014757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56100f55a0 0x7f56101091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:17.539 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.551+0000 7f5615832700 1 --2- 172.21.15.23:0/3894014757 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f55f805c2a0 0x7f55f805e760 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:17.539 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.551+0000 7f5615832700 1 --2- 172.21.15.23:0/3894014757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56100f6840 0x7f56101096b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.551+0000 7f5615832700 1 -- 172.21.15.23:0/3894014757 >> 172.21.15.23:0/3894014757 conn(0x7f56100f0ce0 msgr2=0x7f561005fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.551+0000 7f5615832700 1 -- 172.21.15.23:0/3894014757 shutdown_connections 2021-11-19T08:45:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:17.551+0000 7f5615832700 1 -- 172.21.15.23:0/3894014757 wait complete. 2021-11-19T08:45:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 95 2021-11-19T08:45:17.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:45:15.258264+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24565, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 94, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.134:6825/955716436', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 955716436}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 955716436}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:17.550 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:resolve 2021-11-19T08:45:18.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:18 smithi134 conmon[36363]: cluster 2021-11-19T08:45:16.903461+0000 mgr.smithi023.ehlpcl (mgr.14164) 338 : cluster [DBG] pgmap v316: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-19T08:45:18.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:18 smithi134 conmon[36363]: 2021-11-19T08:45:18.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:18 smithi134 conmon[36363]: audit 2021-11-19T08:45:17.551783+0000 mon.smithi023 (mon.0) 919 : audit [DBG] from='client.? 172.21.15.23:0/3894014757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:18.551 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:18.580 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:18 smithi023 conmon[33490]: cluster 2021-11-19T08:45:16.903461+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:45:18.581 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:18 smithi023 conmon[33490]: 338 : cluster [DBG] pgmap v316: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-19T08:45:18.581 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:18 smithi023 conmon[33490]: audit 2021-11-19T08:45:17.551783+0000 mon.smithi023 (mon.0) 919 : audit [DBG] from='client.? 172.21.15.23:0/3894014757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:18.581 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:18 smithi023 conmon[33490]: 2021-11-19T08:45:18.687 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.698+0000 7f59ea13f700 1 -- 172.21.15.23:0/1116366661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e40f5580 msgr2=0x7f59e40f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:18.687 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.698+0000 7f59ea13f700 1 --2- 172.21.15.23:0/1116366661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e40f5580 0x7f59e40f59a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f59d4004660 tx=0x7f59d4009b30).stop 2021-11-19T08:45:18.687 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.698+0000 7f59ea13f700 1 -- 172.21.15.23:0/1116366661 shutdown_connections 2021-11-19T08:45:18.688 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.698+0000 7f59ea13f700 1 --2- 172.21.15.23:0/1116366661 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59e40f6820 0x7f59e40f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:18.688 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.698+0000 7f59ea13f700 1 --2- 172.21.15.23:0/1116366661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e40f5580 0x7f59e40f59a0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:18.688 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.698+0000 7f59ea13f700 1 -- 172.21.15.23:0/1116366661 >> 172.21.15.23:0/1116366661 conn(0x7f59e40f0ce0 msgr2=0x7f59e40f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:18.688 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.699+0000 7f59ea13f700 1 -- 172.21.15.23:0/1116366661 shutdown_connections 2021-11-19T08:45:18.689 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.699+0000 7f59ea13f700 1 -- 172.21.15.23:0/1116366661 wait complete. 2021-11-19T08:45:18.689 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.699+0000 7f59ea13f700 1 Processor -- start 2021-11-19T08:45:18.689 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.699+0000 7f59ea13f700 1 -- start start 2021-11-19T08:45:18.689 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.699+0000 7f59ea13f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59e40f5580 0x7f59e4106e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:18.690 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.699+0000 7f59ea13f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e40f6820 0x7f59e4107310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:18.690 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.699+0000 7f59ea13f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e41078c0 con 0x7f59e40f6820 2021-11-19T08:45:18.690 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.699+0000 7f59ea13f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e4107a00 con 0x7f59e40f5580 2021-11-19T08:45:18.691 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.700+0000 7f59e2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e40f6820 0x7f59e4107310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:18.691 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.700+0000 7f59e2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e40f6820 0x7f59e4107310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48920/0 (socket says 172.21.15.23:48920) 2021-11-19T08:45:18.691 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.700+0000 7f59e2ffd700 1 -- 172.21.15.23:0/3419535417 learned_addr learned my addr 172.21.15.23:0/3419535417 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:18.692 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.700+0000 7f59e37fe700 1 --2- 172.21.15.23:0/3419535417 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59e40f5580 0x7f59e4106e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:18.692 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.700+0000 7f59e2ffd700 1 -- 172.21.15.23:0/3419535417 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59e40f5580 msgr2=0x7f59e4106e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:18.692 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.700+0000 7f59e2ffd700 1 --2- 172.21.15.23:0/3419535417 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59e40f5580 0x7f59e4106e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:18.692 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.700+0000 7f59e2ffd700 1 -- 172.21.15.23:0/3419535417 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d4005040 con 0x7f59e40f6820 2021-11-19T08:45:18.693 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.700+0000 7f59e2ffd700 1 --2- 172.21.15.23:0/3419535417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e40f6820 0x7f59e4107310 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f59d800e7f0 tx=0x7f59d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:18.693 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.700+0000 7f59cbfff700 1 -- 172.21.15.23:0/3419535417 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59d80092e0 con 0x7f59e40f6820 2021-11-19T08:45:18.693 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.700+0000 7f59ea13f700 1 -- 172.21.15.23:0/3419535417 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e410c4a0 con 0x7f59e40f6820 2021-11-19T08:45:18.693 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.700+0000 7f59cbfff700 1 -- 172.21.15.23:0/3419535417 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f59d800f040 con 0x7f59e40f6820 2021-11-19T08:45:18.693 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.700+0000 7f59cbfff700 1 -- 172.21.15.23:0/3419535417 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59d8014750 con 0x7f59e40f6820 2021-11-19T08:45:18.694 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.700+0000 7f59ea13f700 1 -- 172.21.15.23:0/3419535417 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e410ca60 con 0x7f59e40f6820 2021-11-19T08:45:18.694 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.701+0000 7f59ea13f700 1 -- 172.21.15.23:0/3419535417 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e40fdd10 con 0x7f59e40f6820 2021-11-19T08:45:18.694 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.702+0000 7f59cbfff700 1 -- 172.21.15.23:0/3419535417 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f59d801b030 con 0x7f59e40f6820 2021-11-19T08:45:18.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.702+0000 7f59cbfff700 1 --2- 172.21.15.23:0/3419535417 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f59cc05c2a0 0x7f59cc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:18.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.702+0000 7f59cbfff700 1 -- 172.21.15.23:0/3419535417 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f59d8076280 con 0x7f59e40f6820 2021-11-19T08:45:18.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.702+0000 7f59e37fe700 1 --2- 172.21.15.23:0/3419535417 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f59cc05c2a0 0x7f59cc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:18.696 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.702+0000 7f59e37fe700 1 --2- 172.21.15.23:0/3419535417 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f59cc05c2a0 0x7f59cc05e760 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f59d4013530 tx=0x7f59d4016040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:18.697 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.706+0000 7f59cbfff700 1 -- 172.21.15.23:0/3419535417 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59d80489e0 con 0x7f59e40f6820 2021-11-19T08:45:18.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.901+0000 7f59ea13f700 1 -- 172.21.15.23:0/3419535417 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59e410d100 con 0x7f59e40f6820 2021-11-19T08:45:18.887 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.901+0000 7f59cbfff700 1 -- 172.21.15.23:0/3419535417 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2577 (secure 0 0 0) 0x7f59d8019090 con 0x7f59e40f6820 2021-11-19T08:45:18.888 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:18.889 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":95,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:45:15.258264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24565,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24565":{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":94,"state":"up:resolve","state_seq":19,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:18.890 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.903+0000 7f59ea13f700 1 -- 172.21.15.23:0/3419535417 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f59cc05c2a0 msgr2=0x7f59cc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:18.890 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.903+0000 7f59ea13f700 1 --2- 172.21.15.23:0/3419535417 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f59cc05c2a0 0x7f59cc05e760 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f59d4013530 tx=0x7f59d4016040).stop 2021-11-19T08:45:18.890 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.903+0000 7f59ea13f700 1 -- 172.21.15.23:0/3419535417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e40f6820 msgr2=0x7f59e4107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:18.890 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.903+0000 7f59ea13f700 1 --2- 172.21.15.23:0/3419535417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e40f6820 0x7f59e4107310 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f59d800e7f0 tx=0x7f59d800b040).stop 2021-11-19T08:45:18.891 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.903+0000 7f59ea13f700 1 -- 172.21.15.23:0/3419535417 shutdown_connections 2021-11-19T08:45:18.891 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.903+0000 7f59ea13f700 1 --2- 172.21.15.23:0/3419535417 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59e40f5580 0x7f59e4106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:18.891 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.903+0000 7f59ea13f700 1 --2- 172.21.15.23:0/3419535417 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f59cc05c2a0 0x7f59cc05e760 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:18.891 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.903+0000 7f59ea13f700 1 --2- 172.21.15.23:0/3419535417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e40f6820 0x7f59e4107310 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:18.892 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.903+0000 7f59ea13f700 1 -- 172.21.15.23:0/3419535417 >> 172.21.15.23:0/3419535417 conn(0x7f59e40f0ce0 msgr2=0x7f59e4060240 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:18.892 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.903+0000 7f59ea13f700 1 -- 172.21.15.23:0/3419535417 shutdown_connections 2021-11-19T08:45:18.892 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:18.903+0000 7f59ea13f700 1 -- 172.21.15.23:0/3419535417 wait complete. 2021-11-19T08:45:18.892 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 95 2021-11-19T08:45:18.901 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:45:15.258264+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24565, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 94, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.134:6825/955716436', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 955716436}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 955716436}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:18.902 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:resolve 2021-11-19T08:45:19.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:19 smithi134 conmon[36363]: audit 2021-11-19T08:45:18.903400 2021-11-19T08:45:19.541 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:19 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 920 : audit [DBG] from='client.? 172.21.15.23:0/3419535417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:19.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:19 smithi023 conmon[33490]: audit 2021-11-19T08:45:18.903400+0000 mon.smithi023 (mon.0) 920 : audit [DBG] from='client.? 172.21.15.23:0/3419535417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:19.902 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:20.037 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.049+0000 7f2650fd6700 1 -- 172.21.15.23:0/1547958100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f264c0f6700 msgr2=0x7f264c0f6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:20.038 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.049+0000 7f2650fd6700 1 --2- 172.21.15.23:0/1547958100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f264c0f6700 0x7f264c0f6b60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f2640004660 tx=0x7f2640009b30).stop 2021-11-19T08:45:20.038 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.049+0000 7f2650fd6700 1 -- 172.21.15.23:0/1547958100 shutdown_connections 2021-11-19T08:45:20.038 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.049+0000 7f2650fd6700 1 --2- 172.21.15.23:0/1547958100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f264c0f6700 0x7f264c0f6b60 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:20.039 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.049+0000 7f2650fd6700 1 --2- 172.21.15.23:0/1547958100 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f264c0f5460 0x7f264c0f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:20.039 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.049+0000 7f2650fd6700 1 -- 172.21.15.23:0/1547958100 >> 172.21.15.23:0/1547958100 conn(0x7f264c0f0ba0 msgr2=0x7f264c0f2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:20.039 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.049+0000 7f2650fd6700 1 -- 172.21.15.23:0/1547958100 shutdown_connections 2021-11-19T08:45:20.039 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.050+0000 7f2650fd6700 1 -- 172.21.15.23:0/1547958100 wait complete. 2021-11-19T08:45:20.040 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.050+0000 7f2650fd6700 1 Processor -- start 2021-11-19T08:45:20.040 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.050+0000 7f2650fd6700 1 -- start start 2021-11-19T08:45:20.040 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f2650fd6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f264c0f5460 0x7f264c106d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:20.040 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f2650fd6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f264c0f6700 0x7f264c107280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:20.040 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f2650fd6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264c107880 con 0x7f264c0f6700 2021-11-19T08:45:20.041 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f2650fd6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264c108090 con 0x7f264c0f5460 2021-11-19T08:45:20.041 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f264affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f264c0f6700 0x7f264c107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:20.041 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f264affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f264c0f6700 0x7f264c107280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48928/0 (socket says 172.21.15.23:48928) 2021-11-19T08:45:20.041 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f264affd700 1 -- 172.21.15.23:0/3448159119 learned_addr learned my addr 172.21.15.23:0/3448159119 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:20.042 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f264b7fe700 1 --2- 172.21.15.23:0/3448159119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f264c0f5460 0x7f264c106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:20.042 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f264affd700 1 -- 172.21.15.23:0/3448159119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f264c0f5460 msgr2=0x7f264c106d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:20.042 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f264affd700 1 --2- 172.21.15.23:0/3448159119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f264c0f5460 0x7f264c106d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:20.042 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f264affd700 1 -- 172.21.15.23:0/3448159119 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2640005040 con 0x7f264c0f6700 2021-11-19T08:45:20.043 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f264affd700 1 --2- 172.21.15.23:0/3448159119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f264c0f6700 0x7f264c107280 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f2640009f50 tx=0x7f2640009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:20.043 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f2633fff700 1 -- 172.21.15.23:0/3448159119 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2640018070 con 0x7f264c0f6700 2021-11-19T08:45:20.043 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f2633fff700 1 -- 172.21.15.23:0/3448159119 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2640013410 con 0x7f264c0f6700 2021-11-19T08:45:20.043 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f2650fd6700 1 -- 172.21.15.23:0/3448159119 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f264c1082b0 con 0x7f264c0f6700 2021-11-19T08:45:20.043 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f2633fff700 1 -- 172.21.15.23:0/3448159119 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f264001c3c0 con 0x7f264c0f6700 2021-11-19T08:45:20.044 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.051+0000 7f2650fd6700 1 -- 172.21.15.23:0/3448159119 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264c108810 con 0x7f264c0f6700 2021-11-19T08:45:20.045 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.052+0000 7f2650fd6700 1 -- 172.21.15.23:0/3448159119 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f264c0f4850 con 0x7f264c0f6700 2021-11-19T08:45:20.046 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.053+0000 7f2633fff700 1 -- 172.21.15.23:0/3448159119 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f264001d420 con 0x7f264c0f6700 2021-11-19T08:45:20.046 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.053+0000 7f2633fff700 1 --2- 172.21.15.23:0/3448159119 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f263405c300 0x7f263405e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:20.046 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.053+0000 7f2633fff700 1 -- 172.21.15.23:0/3448159119 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f2640078170 con 0x7f264c0f6700 2021-11-19T08:45:20.046 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.053+0000 7f264b7fe700 1 --2- 172.21.15.23:0/3448159119 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f263405c300 0x7f263405e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:20.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.054+0000 7f264b7fe700 1 --2- 172.21.15.23:0/3448159119 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f263405c300 0x7f263405e7c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f264c106b00 tx=0x7f263c009c00).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:20.047 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.057+0000 7f2633fff700 1 -- 172.21.15.23:0/3448159119 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f264004a880 con 0x7f264c0f6700 2021-11-19T08:45:20.237 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.252+0000 7f2650fd6700 1 -- 172.21.15.23:0/3448159119 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f264c045b20 con 0x7f264c0f6700 2021-11-19T08:45:20.238 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.252+0000 7f2633fff700 1 -- 172.21.15.23:0/3448159119 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2577 (secure 0 0 0) 0x7f264004dea0 con 0x7f264c0f6700 2021-11-19T08:45:20.238 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:20.239 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":95,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:45:15.258264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24565,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24565":{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":94,"state":"up:resolve","state_seq":19,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:20.240 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.254+0000 7f2650fd6700 1 -- 172.21.15.23:0/3448159119 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f263405c300 msgr2=0x7f263405e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:20.240 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.254+0000 7f2650fd6700 1 --2- 172.21.15.23:0/3448159119 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f263405c300 0x7f263405e7c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f264c106b00 tx=0x7f263c009c00).stop 2021-11-19T08:45:20.240 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.254+0000 7f2650fd6700 1 -- 172.21.15.23:0/3448159119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f264c0f6700 msgr2=0x7f264c107280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:20.241 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.254+0000 7f2650fd6700 1 --2- 172.21.15.23:0/3448159119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f264c0f6700 0x7f264c107280 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f2640009f50 tx=0x7f2640009b30).stop 2021-11-19T08:45:20.241 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.254+0000 7f2650fd6700 1 -- 172.21.15.23:0/3448159119 shutdown_connections 2021-11-19T08:45:20.241 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.254+0000 7f2650fd6700 1 --2- 172.21.15.23:0/3448159119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f264c0f5460 0x7f264c106d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:20.241 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.254+0000 7f2650fd6700 1 --2- 172.21.15.23:0/3448159119 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f263405c300 0x7f263405e7c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:20.242 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.254+0000 7f2650fd6700 1 --2- 172.21.15.23:0/3448159119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f264c0f6700 0x7f264c107280 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:20.242 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.254+0000 7f2650fd6700 1 -- 172.21.15.23:0/3448159119 >> 172.21.15.23:0/3448159119 conn(0x7f264c0f0ba0 msgr2=0x7f264c05f450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:20.242 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.254+0000 7f2650fd6700 1 -- 172.21.15.23:0/3448159119 shutdown_connections 2021-11-19T08:45:20.242 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:20.254+0000 7f2650fd6700 1 -- 172.21.15.23:0/3448159119 wait complete. 2021-11-19T08:45:20.243 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 95 2021-11-19T08:45:20.251 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:45:15.258264+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24565, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 94, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.134:6825/955716436', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 955716436}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 955716436}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:20.252 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:resolve 2021-11-19T08:45:20.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:20 smithi134 conmon[36363]: cluster 2021-11-19T08:45:18.903858+0000 mgr.smithi023.ehlpcl (mgr.14164) 339 2021-11-19T08:45:20.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:20 smithi134 conmon[36363]: : cluster [DBG] pgmap v317: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-19T08:45:20.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:20 smithi134 conmon[36363]: audit 2021-11-19T08:45:20.254338+0000 mon.smithi023 (mon.0) 921 : audit [DBG] from='client.? 172.21.15.23:0/3448159119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:20.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:20 smithi023 conmon[33490]: cluster 2021-11-19T08:45:18.903858+0000 mgr.smithi023.ehlpcl (mgr.14164) 339 : cluster [DBG] pgmap v317: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-19T08:45:20.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:20 smithi023 conmon[33490]: audit 2021-11-19T08:45:20.254338+0000 mon.smithi023 (mon.0) 921 : audit [DBG] from='client.? 172.21.15.23:0/3448159119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:21.253 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:21.388 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.400+0000 7f062b3bd700 1 -- 172.21.15.23:0/590503418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06240f5580 msgr2=0x7f06240f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:21.388 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.400+0000 7f062b3bd700 1 --2- 172.21.15.23:0/590503418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06240f5580 0x7f06240f59a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f061c004660 tx=0x7f061c009b30).stop 2021-11-19T08:45:21.388 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.400+0000 7f062b3bd700 1 -- 172.21.15.23:0/590503418 shutdown_connections 2021-11-19T08:45:21.389 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.400+0000 7f062b3bd700 1 --2- 172.21.15.23:0/590503418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06240f6820 0x7f06240f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:21.389 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.400+0000 7f062b3bd700 1 --2- 172.21.15.23:0/590503418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06240f5580 0x7f06240f59a0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:21.389 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.400+0000 7f062b3bd700 1 -- 172.21.15.23:0/590503418 >> 172.21.15.23:0/590503418 conn(0x7f06240f0ce0 msgr2=0x7f06240f3100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:21.389 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.400+0000 7f062b3bd700 1 -- 172.21.15.23:0/590503418 shutdown_connections 2021-11-19T08:45:21.390 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.400+0000 7f062b3bd700 1 -- 172.21.15.23:0/590503418 wait complete. 2021-11-19T08:45:21.390 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.400+0000 7f062b3bd700 1 Processor -- start 2021-11-19T08:45:21.390 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.400+0000 7f062b3bd700 1 -- start start 2021-11-19T08:45:21.390 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f062b3bd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06240f5580 0x7f06240604b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:21.391 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f062b3bd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06240f6820 0x7f062410bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:21.391 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f062b3bd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062410c1b0 con 0x7f06240f6820 2021-11-19T08:45:21.391 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f062b3bd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062410c2f0 con 0x7f06240f5580 2021-11-19T08:45:21.391 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f0628958700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06240f6820 0x7f062410bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:21.391 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f0628958700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06240f6820 0x7f062410bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48938/0 (socket says 172.21.15.23:48938) 2021-11-19T08:45:21.392 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f0628958700 1 -- 172.21.15.23:0/1796764940 learned_addr learned my addr 172.21.15.23:0/1796764940 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:21.392 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f0629159700 1 --2- 172.21.15.23:0/1796764940 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06240f5580 0x7f06240604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:21.392 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f0628958700 1 -- 172.21.15.23:0/1796764940 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06240f5580 msgr2=0x7f06240604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:21.392 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f0628958700 1 --2- 172.21.15.23:0/1796764940 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06240f5580 0x7f06240604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:21.393 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f0628958700 1 -- 172.21.15.23:0/1796764940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061c005040 con 0x7f06240f6820 2021-11-19T08:45:21.393 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f0629159700 1 --2- 172.21.15.23:0/1796764940 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06240f5580 0x7f06240604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:45:21.393 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f0628958700 1 --2- 172.21.15.23:0/1796764940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06240f6820 0x7f062410bcc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f06180026d0 tx=0x7f0618009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:21.393 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f0621ffb700 1 -- 172.21.15.23:0/1796764940 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0618009320 con 0x7f06240f6820 2021-11-19T08:45:21.393 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f0621ffb700 1 -- 172.21.15.23:0/1796764940 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f061800f040 con 0x7f06240f6820 2021-11-19T08:45:21.394 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f062b3bd700 1 -- 172.21.15.23:0/1796764940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f062410c550 con 0x7f06240f6820 2021-11-19T08:45:21.394 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.401+0000 7f0621ffb700 1 -- 172.21.15.23:0/1796764940 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06180147e0 con 0x7f06240f6820 2021-11-19T08:45:21.394 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.402+0000 7f062b3bd700 1 -- 172.21.15.23:0/1796764940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f062410cae0 con 0x7f06240f6820 2021-11-19T08:45:21.394 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.402+0000 7f062b3bd700 1 -- 172.21.15.23:0/1796764940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0624046330 con 0x7f06240f6820 2021-11-19T08:45:21.395 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.403+0000 7f0621ffb700 1 -- 172.21.15.23:0/1796764940 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f06180077b0 con 0x7f06240f6820 2021-11-19T08:45:21.395 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.403+0000 7f0621ffb700 1 --2- 172.21.15.23:0/1796764940 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f060c05c2a0 0x7f060c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:21.396 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.403+0000 7f0621ffb700 1 -- 172.21.15.23:0/1796764940 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f0618076b40 con 0x7f06240f6820 2021-11-19T08:45:21.396 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.403+0000 7f0629159700 1 --2- 172.21.15.23:0/1796764940 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f060c05c2a0 0x7f060c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:21.396 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.403+0000 7f0629159700 1 --2- 172.21.15.23:0/1796764940 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f060c05c2a0 0x7f060c05e760 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f061c009d60 tx=0x7f061c009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:21.396 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.407+0000 7f0621ffb700 1 -- 172.21.15.23:0/1796764940 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0618049210 con 0x7f06240f6820 2021-11-19T08:45:21.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:21 smithi134 conmon[36363]: audit 2021-11-19T08:45:20.922505+0000 mon.smithi023 (mon.0) 922 : 2021-11-19T08:45:21.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:21 smithi134 conmon[36363]: audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:45:21.540 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:21 smithi134 conmon[36363]: audit 2021-11-19T08:45:20.923839+0000 mon.smithi023 (mon.0) 923 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:45:21.589 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.603+0000 7f062b3bd700 1 -- 172.21.15.23:0/1796764940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0624027a00 con 0x7f06240f6820 2021-11-19T08:45:21.590 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.604+0000 7f0621ffb700 1 -- 172.21.15.23:0/1796764940 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2577 (secure 0 0 0) 0x7f061804c830 con 0x7f06240f6820 2021-11-19T08:45:21.590 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:21.591 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":95,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:45:15.258264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24565,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24565":{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":94,"state":"up:resolve","state_seq":19,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:21.592 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.606+0000 7f062b3bd700 1 -- 172.21.15.23:0/1796764940 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f060c05c2a0 msgr2=0x7f060c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:21.592 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.606+0000 7f062b3bd700 1 --2- 172.21.15.23:0/1796764940 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f060c05c2a0 0x7f060c05e760 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f061c009d60 tx=0x7f061c009b30).stop 2021-11-19T08:45:21.592 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.606+0000 7f062b3bd700 1 -- 172.21.15.23:0/1796764940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06240f6820 msgr2=0x7f062410bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:21.594 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.606+0000 7f062b3bd700 1 --2- 172.21.15.23:0/1796764940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06240f6820 0x7f062410bcc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f06180026d0 tx=0x7f0618009de0).stop 2021-11-19T08:45:21.594 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.606+0000 7f062b3bd700 1 -- 172.21.15.23:0/1796764940 shutdown_connections 2021-11-19T08:45:21.594 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.606+0000 7f062b3bd700 1 --2- 172.21.15.23:0/1796764940 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06240f5580 0x7f06240604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:21.594 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.606+0000 7f062b3bd700 1 --2- 172.21.15.23:0/1796764940 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f060c05c2a0 0x7f060c05e760 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:21.595 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.606+0000 7f062b3bd700 1 --2- 172.21.15.23:0/1796764940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06240f6820 0x7f062410bcc0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:21.595 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.606+0000 7f062b3bd700 1 -- 172.21.15.23:0/1796764940 >> 172.21.15.23:0/1796764940 conn(0x7f06240f0ce0 msgr2=0x7f06240fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:21.595 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.606+0000 7f062b3bd700 1 -- 172.21.15.23:0/1796764940 shutdown_connections 2021-11-19T08:45:21.595 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:21.606+0000 7f062b3bd700 1 -- 172.21.15.23:0/1796764940 wait complete. 2021-11-19T08:45:21.596 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 95 2021-11-19T08:45:21.602 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:21 smithi023 conmon[33490]: audit 2021-11-19T08:45:20.922505+0000 mon.smithi023 (mon.0) 922 : audit 2021-11-19T08:45:21.602 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:21 smithi023 conmon[33490]: [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/mirror_snapshot_schedule"}]: dispatch 2021-11-19T08:45:21.603 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:21 smithi023 conmon[33490]: audit 2021-11-19T08:45:20. 2021-11-19T08:45:21.603 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:21 smithi023 conmon[33490]: 923839+0000 mon.smithi023 (mon.0) 923 : audit [INF] from='mgr.14164 172.21.15.23:0/3085710151' entity='mgr.smithi023.ehlpcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.ehlpcl/trash_purge_schedule"}]: dispatch 2021-11-19T08:45:21.604 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:45:15.258264+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24565, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 94, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.134:6825/955716436', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 955716436}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 955716436}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:21.605 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:resolve 2021-11-19T08:45:22.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:22 smithi134 conmon[36363]: cluster 2021-11-19T08:45:20.904326+0000 mgr.smithi023.ehlpcl (mgr.14164) 2021-11-19T08:45:22.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:22 smithi134 conmon[36363]: 340 : cluster [DBG] pgmap v318: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:45:22.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:22 smithi134 conmon[36363]: audit 2021-11-19T08:45:21.606070+0000 mon.smithi023 (mon.0) 924 : audit [DBG] from='client.? 172.21.15.23:0/1796764940' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:22.606 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:22.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:22 smithi023 conmon[33490]: cluster 2021-11-19T08:45:20.904326+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:45:22.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:22 smithi023 conmon[33490]: mgr.14164) 340 : cluster [DBG] pgmap v318: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:45:22.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:22 smithi023 conmon[33490]: audit 2021-11-19T08:45:21.606070+0000 mon.smithi023 (mon. 2021-11-19T08:45:22.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:22 smithi023 conmon[33490]: 0) 924 : audit [DBG] from='client.? 172.21.15.23:0/1796764940' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36359e700 1 -- 172.21.15.23:0/558543487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3640f7870 msgr2=0x7fc3640f7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:22.748 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36359e700 1 --2- 172.21.15.23:0/558543487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3640f7870 0x7fc3640f7cd0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7fc358004660 tx=0x7fc358009b30).stop 2021-11-19T08:45:22.748 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36359e700 1 -- 172.21.15.23:0/558543487 shutdown_connections 2021-11-19T08:45:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36359e700 1 --2- 172.21.15.23:0/558543487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3640f7870 0x7fc3640f7cd0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36359e700 1 --2- 172.21.15.23:0/558543487 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3640f65d0 0x7fc3640f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:22.750 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36359e700 1 -- 172.21.15.23:0/558543487 >> 172.21.15.23:0/558543487 conn(0x7fc3640f1d10 msgr2=0x7fc3640f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:22.750 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36359e700 1 -- 172.21.15.23:0/558543487 shutdown_connections 2021-11-19T08:45:22.751 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36359e700 1 -- 172.21.15.23:0/558543487 wait complete. 2021-11-19T08:45:22.751 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36359e700 1 Processor -- start 2021-11-19T08:45:22.751 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36359e700 1 -- start start 2021-11-19T08:45:22.751 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36359e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3640f65d0 0x7fc36406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:22.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36359e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3640f7870 0x7fc364069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:22.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36359e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36406a380 con 0x7fc3640f65d0 2021-11-19T08:45:22.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36359e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36406a4c0 con 0x7fc3640f7870 2021-11-19T08:45:22.752 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.760+0000 7fc36259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3640f65d0 0x7fc36406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:22.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.761+0000 7fc36259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3640f65d0 0x7fc36406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48948/0 (socket says 172.21.15.23:48948) 2021-11-19T08:45:22.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.761+0000 7fc36259c700 1 -- 172.21.15.23:0/2025853848 learned_addr learned my addr 172.21.15.23:0/2025853848 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:22.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.761+0000 7fc361d9b700 1 --2- 172.21.15.23:0/2025853848 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3640f7870 0x7fc364069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:22.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.761+0000 7fc36259c700 1 -- 172.21.15.23:0/2025853848 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3640f7870 msgr2=0x7fc364069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:22.753 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.761+0000 7fc36259c700 1 --2- 172.21.15.23:0/2025853848 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3640f7870 0x7fc364069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:22.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.761+0000 7fc36259c700 1 -- 172.21.15.23:0/2025853848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc358005040 con 0x7fc3640f65d0 2021-11-19T08:45:22.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.761+0000 7fc36259c700 1 --2- 172.21.15.23:0/2025853848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3640f65d0 0x7fc36406b9a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fc35400c530 tx=0x7fc354007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:22.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.761+0000 7fc352ffd700 1 -- 172.21.15.23:0/2025853848 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc35400f650 con 0x7fc3640f65d0 2021-11-19T08:45:22.754 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.761+0000 7fc352ffd700 1 -- 172.21.15.23:0/2025853848 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc35400fc60 con 0x7fc3640f65d0 2021-11-19T08:45:22.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.761+0000 7fc352ffd700 1 -- 172.21.15.23:0/2025853848 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc354012d20 con 0x7fc3640f65d0 2021-11-19T08:45:22.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.761+0000 7fc36359e700 1 -- 172.21.15.23:0/2025853848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36406a7a0 con 0x7fc3640f65d0 2021-11-19T08:45:22.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.761+0000 7fc36359e700 1 -- 172.21.15.23:0/2025853848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36410ed50 con 0x7fc3640f65d0 2021-11-19T08:45:22.755 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.762+0000 7fc36359e700 1 -- 172.21.15.23:0/2025853848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc364045ad0 con 0x7fc3640f65d0 2021-11-19T08:45:22.757 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.762+0000 7fc352ffd700 1 -- 172.21.15.23:0/2025853848 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc354019030 con 0x7fc3640f65d0 2021-11-19T08:45:22.757 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.763+0000 7fc352ffd700 1 --2- 172.21.15.23:0/2025853848 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc34c05c300 0x7fc34c05e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:22.757 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.763+0000 7fc352ffd700 1 -- 172.21.15.23:0/2025853848 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7fc354076120 con 0x7fc3640f65d0 2021-11-19T08:45:22.757 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.763+0000 7fc361d9b700 1 --2- 172.21.15.23:0/2025853848 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc34c05c300 0x7fc34c05e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:22.758 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.763+0000 7fc361d9b700 1 --2- 172.21.15.23:0/2025853848 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc34c05c300 0x7fc34c05e7c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc358009d60 tx=0x7fc358009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:22.758 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.767+0000 7fc352ffd700 1 -- 172.21.15.23:0/2025853848 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc354048880 con 0x7fc3640f65d0 2021-11-19T08:45:22.947 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.962+0000 7fc36359e700 1 -- 172.21.15.23:0/2025853848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc364027a00 con 0x7fc3640f65d0 2021-11-19T08:45:22.948 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.962+0000 7fc352ffd700 1 -- 172.21.15.23:0/2025853848 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2577 (secure 0 0 0) 0x7fc354017020 con 0x7fc3640f65d0 2021-11-19T08:45:22.948 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:22.949 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":95,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:45:15.258264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24565,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24565":{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":94,"state":"up:resolve","state_seq":19,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:22.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.964+0000 7fc36359e700 1 -- 172.21.15.23:0/2025853848 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc34c05c300 msgr2=0x7fc34c05e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:22.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.964+0000 7fc36359e700 1 --2- 172.21.15.23:0/2025853848 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc34c05c300 0x7fc34c05e7c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc358009d60 tx=0x7fc358009b30).stop 2021-11-19T08:45:22.951 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.964+0000 7fc36359e700 1 -- 172.21.15.23:0/2025853848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3640f65d0 msgr2=0x7fc36406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:22.951 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.964+0000 7fc36359e700 1 --2- 172.21.15.23:0/2025853848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3640f65d0 0x7fc36406b9a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fc35400c530 tx=0x7fc354007e10).stop 2021-11-19T08:45:22.951 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.964+0000 7fc36359e700 1 -- 172.21.15.23:0/2025853848 shutdown_connections 2021-11-19T08:45:22.952 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.964+0000 7fc36359e700 1 --2- 172.21.15.23:0/2025853848 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fc34c05c300 0x7fc34c05e7c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:22.952 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.964+0000 7fc36359e700 1 --2- 172.21.15.23:0/2025853848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3640f65d0 0x7fc36406b9a0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:22.952 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.964+0000 7fc36359e700 1 --2- 172.21.15.23:0/2025853848 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3640f7870 0x7fc364069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:22.953 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.964+0000 7fc36359e700 1 -- 172.21.15.23:0/2025853848 >> 172.21.15.23:0/2025853848 conn(0x7fc3640f1d10 msgr2=0x7fc3640fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:22.953 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.964+0000 7fc36359e700 1 -- 172.21.15.23:0/2025853848 shutdown_connections 2021-11-19T08:45:22.953 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:22.964+0000 7fc36359e700 1 -- 172.21.15.23:0/2025853848 wait complete. 2021-11-19T08:45:22.953 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 95 2021-11-19T08:45:22.962 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:45:15.258264+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24565, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 94, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.134:6825/955716436', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 955716436}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 955716436}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:22.962 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:resolve 2021-11-19T08:45:23.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:23 smithi023 conmon[33490]: audit 2021-11-19T08:45:22.964310+0000 mon.smithi023 ( 2021-11-19T08:45:23.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:23 smithi023 conmon[33490]: mon.0) 925 : audit [DBG] from='client.? 172.21.15.23:0/2025853848' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:23.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:23 smithi134 conmon[36363]: audit 2021-11-19T08:45:22.964310+0000 mon.smithi023 (mon.0) 925 : audit [DBG] from='client.? 172.21.15.23:0/2025853848' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:23.963 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.110+0000 7f3738984700 1 -- 172.21.15.23:0/2305073624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37340f5500 msgr2=0x7f37340f5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.110+0000 7f3738984700 1 --2- 172.21.15.23:0/2305073624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37340f5500 0x7f37340f5920 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f3724004660 tx=0x7f3724009b30).stop 2021-11-19T08:45:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.110+0000 7f3738984700 1 -- 172.21.15.23:0/2305073624 shutdown_connections 2021-11-19T08:45:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.110+0000 7f3738984700 1 --2- 172.21.15.23:0/2305073624 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37340f67a0 0x7f37340f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.110+0000 7f3738984700 1 --2- 172.21.15.23:0/2305073624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37340f5500 0x7f37340f5920 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.110+0000 7f3738984700 1 -- 172.21.15.23:0/2305073624 >> 172.21.15.23:0/2305073624 conn(0x7f37340f0c80 msgr2=0x7f37340f30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.110+0000 7f3738984700 1 -- 172.21.15.23:0/2305073624 shutdown_connections 2021-11-19T08:45:24.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.110+0000 7f3738984700 1 -- 172.21.15.23:0/2305073624 wait complete. 2021-11-19T08:45:24.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.111+0000 7f3738984700 1 Processor -- start 2021-11-19T08:45:24.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.111+0000 7f3738984700 1 -- start start 2021-11-19T08:45:24.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.111+0000 7f3738984700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37340f5500 0x7f3734105d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:24.100 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.111+0000 7f3738984700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37340f67a0 0x7f3734106230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.111+0000 7f3738984700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37341067e0 con 0x7f37340f67a0 2021-11-19T08:45:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.111+0000 7f3738984700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3734106920 con 0x7f37340f5500 2021-11-19T08:45:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.111+0000 7f3731d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37340f67a0 0x7f3734106230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.111+0000 7f3731d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37340f67a0 0x7f3734106230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48958/0 (socket says 172.21.15.23:48958) 2021-11-19T08:45:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.111+0000 7f3731d9b700 1 -- 172.21.15.23:0/3891586006 learned_addr learned my addr 172.21.15.23:0/3891586006 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.111+0000 7f373259c700 1 --2- 172.21.15.23:0/3891586006 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37340f5500 0x7f3734105d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.112+0000 7f3731d9b700 1 -- 172.21.15.23:0/3891586006 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37340f5500 msgr2=0x7f3734105d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.112+0000 7f3731d9b700 1 --2- 172.21.15.23:0/3891586006 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37340f5500 0x7f3734105d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:24.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.112+0000 7f3731d9b700 1 -- 172.21.15.23:0/3891586006 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3724005040 con 0x7f37340f67a0 2021-11-19T08:45:24.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.112+0000 7f3731d9b700 1 --2- 172.21.15.23:0/3891586006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37340f67a0 0x7f3734106230 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f371c0026d0 tx=0x7f371c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:24.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.112+0000 7f372affd700 1 -- 172.21.15.23:0/3891586006 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f371c0079c0 con 0x7f37340f67a0 2021-11-19T08:45:24.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.112+0000 7f372affd700 1 -- 172.21.15.23:0/3891586006 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f371c00f040 con 0x7f37340f67a0 2021-11-19T08:45:24.103 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.112+0000 7f3738984700 1 -- 172.21.15.23:0/3891586006 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f373410b3c0 con 0x7f37340f67a0 2021-11-19T08:45:24.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.112+0000 7f372affd700 1 -- 172.21.15.23:0/3891586006 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f371c0148b0 con 0x7f37340f67a0 2021-11-19T08:45:24.104 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.112+0000 7f3738984700 1 -- 172.21.15.23:0/3891586006 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f373410b980 con 0x7f37340f67a0 2021-11-19T08:45:24.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.113+0000 7f3738984700 1 -- 172.21.15.23:0/3891586006 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3734046330 con 0x7f37340f67a0 2021-11-19T08:45:24.105 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.113+0000 7f372affd700 1 -- 172.21.15.23:0/3891586006 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f371c007b20 con 0x7f37340f67a0 2021-11-19T08:45:24.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.114+0000 7f372affd700 1 --2- 172.21.15.23:0/3891586006 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f372005c2b0 0x7f372005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:24.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.114+0000 7f372affd700 1 -- 172.21.15.23:0/3891586006 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f371c076460 con 0x7f37340f67a0 2021-11-19T08:45:24.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.114+0000 7f373259c700 1 --2- 172.21.15.23:0/3891586006 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f372005c2b0 0x7f372005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:24.106 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.114+0000 7f373259c700 1 --2- 172.21.15.23:0/3891586006 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f372005c2b0 0x7f372005e770 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f3724009f50 tx=0x7f3724009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:24.107 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.118+0000 7f372affd700 1 -- 172.21.15.23:0/3891586006 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f371c048b70 con 0x7f37340f67a0 2021-11-19T08:45:24.301 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.315+0000 7f3738984700 1 -- 172.21.15.23:0/3891586006 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3734027a00 con 0x7f37340f67a0 2021-11-19T08:45:24.448 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.316+0000 7f372affd700 1 -- 172.21.15.23:0/3891586006 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2577 (secure 0 0 0) 0x7f371c04c190 con 0x7f37340f67a0 2021-11-19T08:45:24.448 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:24.449 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":95,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:45:15.258264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24565,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24565":{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":94,"state":"up:resolve","state_seq":19,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.318+0000 7f3738984700 1 -- 172.21.15.23:0/3891586006 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f372005c2b0 msgr2=0x7f372005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.318+0000 7f3738984700 1 --2- 172.21.15.23:0/3891586006 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f372005c2b0 0x7f372005e770 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f3724009f50 tx=0x7f3724009b30).stop 2021-11-19T08:45:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.318+0000 7f3738984700 1 -- 172.21.15.23:0/3891586006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37340f67a0 msgr2=0x7f3734106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.318+0000 7f3738984700 1 --2- 172.21.15.23:0/3891586006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37340f67a0 0x7f3734106230 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f371c0026d0 tx=0x7f371c009de0).stop 2021-11-19T08:45:24.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.318+0000 7f3738984700 1 -- 172.21.15.23:0/3891586006 shutdown_connections 2021-11-19T08:45:24.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.318+0000 7f3738984700 1 --2- 172.21.15.23:0/3891586006 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37340f5500 0x7f3734105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:24.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.318+0000 7f3738984700 1 --2- 172.21.15.23:0/3891586006 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f372005c2b0 0x7f372005e770 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:24.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.318+0000 7f3738984700 1 --2- 172.21.15.23:0/3891586006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37340f67a0 0x7f3734106230 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:24.452 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.318+0000 7f3738984700 1 -- 172.21.15.23:0/3891586006 >> 172.21.15.23:0/3891586006 conn(0x7f37340f0c80 msgr2=0x7f37340fa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:24.453 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.318+0000 7f3738984700 1 -- 172.21.15.23:0/3891586006 shutdown_connections 2021-11-19T08:45:24.453 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:24.318+0000 7f3738984700 1 -- 172.21.15.23:0/3891586006 wait complete. 2021-11-19T08:45:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 95 2021-11-19T08:45:24.454 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:45:15.258264+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24565, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 94, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.134:6825/955716436', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 955716436}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 955716436}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:24.455 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:resolve 2021-11-19T08:45:24.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:24 smithi023 conmon[33490]: cluster 2021-11-19T08:45:22.904717+0000 mgr.smithi023.ehlpcl 2021-11-19T08:45:24.642 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:24 smithi023 conmon[33490]: (mgr.14164) 341 : cluster [DBG] pgmap v319: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:45:24.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:24 smithi134 conmon[36363]: cluster 2021-11-19T08:45:22.904717+0000 mgr.smithi023.ehlpcl (mgr.14164) 341 : cluster [DBG] pgmap v319: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-11-19T08:45:25.456 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:25.593 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.603+0000 7f87e4fb0700 1 -- 172.21.15.23:0/2071039312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87e00f6810 msgr2=0x7f87e00f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:25.594 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.603+0000 7f87e4fb0700 1 --2- 172.21.15.23:0/2071039312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87e00f6810 0x7f87e00f6c70 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f87d4004660 tx=0x7f87d4009b30).stop 2021-11-19T08:45:25.594 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.603+0000 7f87e4fb0700 1 -- 172.21.15.23:0/2071039312 shutdown_connections 2021-11-19T08:45:25.594 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.603+0000 7f87e4fb0700 1 --2- 172.21.15.23:0/2071039312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87e00f6810 0x7f87e00f6c70 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:25.595 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.603+0000 7f87e4fb0700 1 --2- 172.21.15.23:0/2071039312 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87e00f5570 0x7f87e00f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:25.595 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.603+0000 7f87e4fb0700 1 -- 172.21.15.23:0/2071039312 >> 172.21.15.23:0/2071039312 conn(0x7f87e00f0cb0 msgr2=0x7f87e00f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:25.596 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.603+0000 7f87e4fb0700 1 -- 172.21.15.23:0/2071039312 shutdown_connections 2021-11-19T08:45:25.596 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.603+0000 7f87e4fb0700 1 -- 172.21.15.23:0/2071039312 wait complete. 2021-11-19T08:45:25.596 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.604+0000 7f87e4fb0700 1 Processor -- start 2021-11-19T08:45:25.596 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.604+0000 7f87e4fb0700 1 -- start start 2021-11-19T08:45:25.597 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.604+0000 7f87e4fb0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87e00f5570 0x7f87e0060590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:25.597 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.604+0000 7f87e4fb0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87e00f6810 0x7f87e010bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:25.597 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.604+0000 7f87e4fb0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e010c160 con 0x7f87e00f6810 2021-11-19T08:45:25.597 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.604+0000 7f87e4fb0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e010c2a0 con 0x7f87e00f5570 2021-11-19T08:45:25.598 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.604+0000 7f87ddd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87e00f6810 0x7f87e010bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:25.598 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.604+0000 7f87ddd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87e00f6810 0x7f87e010bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48968/0 (socket says 172.21.15.23:48968) 2021-11-19T08:45:25.598 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.604+0000 7f87ddd9b700 1 -- 172.21.15.23:0/619033886 learned_addr learned my addr 172.21.15.23:0/619033886 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:25.598 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.604+0000 7f87de59c700 1 --2- 172.21.15.23:0/619033886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87e00f5570 0x7f87e0060590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:25.599 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.605+0000 7f87ddd9b700 1 -- 172.21.15.23:0/619033886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87e00f5570 msgr2=0x7f87e0060590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:25.599 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.605+0000 7f87ddd9b700 1 --2- 172.21.15.23:0/619033886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87e00f5570 0x7f87e0060590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:25.599 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.605+0000 7f87ddd9b700 1 -- 172.21.15.23:0/619033886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d4005040 con 0x7f87e00f6810 2021-11-19T08:45:25.599 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.605+0000 7f87de59c700 1 --2- 172.21.15.23:0/619033886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87e00f5570 0x7f87e0060590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-11-19T08:45:25.600 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.605+0000 7f87ddd9b700 1 --2- 172.21.15.23:0/619033886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87e00f6810 0x7f87e010bc70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f87d4007a70 tx=0x7f87d4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:25.600 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.605+0000 7f87ceffd700 1 -- 172.21.15.23:0/619033886 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87d4018070 con 0x7f87e00f6810 2021-11-19T08:45:25.600 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.605+0000 7f87ceffd700 1 -- 172.21.15.23:0/619033886 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87d4013410 con 0x7f87e00f6810 2021-11-19T08:45:25.600 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.605+0000 7f87ceffd700 1 -- 172.21.15.23:0/619033886 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87d401c620 con 0x7f87e00f6810 2021-11-19T08:45:25.601 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.605+0000 7f87e4fb0700 1 -- 172.21.15.23:0/619033886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e010c4a0 con 0x7f87e00f6810 2021-11-19T08:45:25.601 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.605+0000 7f87e4fb0700 1 -- 172.21.15.23:0/619033886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e010ca00 con 0x7f87e00f6810 2021-11-19T08:45:25.601 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.606+0000 7f87e4fb0700 1 -- 172.21.15.23:0/619033886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e0027a00 con 0x7f87e00f6810 2021-11-19T08:45:25.601 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.606+0000 7f87ceffd700 1 -- 172.21.15.23:0/619033886 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f87d401c8e0 con 0x7f87e00f6810 2021-11-19T08:45:25.602 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.607+0000 7f87ceffd700 1 --2- 172.21.15.23:0/619033886 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f87c805c2a0 0x7f87c805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:25.602 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.607+0000 7f87ceffd700 1 -- 172.21.15.23:0/619033886 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f87d4078390 con 0x7f87e00f6810 2021-11-19T08:45:25.602 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.607+0000 7f87de59c700 1 --2- 172.21.15.23:0/619033886 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f87c805c2a0 0x7f87c805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:25.602 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.607+0000 7f87de59c700 1 --2- 172.21.15.23:0/619033886 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f87c805c2a0 0x7f87c805e760 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f87e00f6670 tx=0x7f87d0009040).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:25.603 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.611+0000 7f87ceffd700 1 -- 172.21.15.23:0/619033886 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87d404aa40 con 0x7f87e00f6810 2021-11-19T08:45:25.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:25 smithi023 conmon[33490]: audit 2021-11-19T08:45:24.317773+0000 mon.smithi023 (mon.0) 926 : 2021-11-19T08:45:25.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:25 smithi023 conmon[33490]: audit [DBG] from='client.? 172.21.15.23:0/3891586006' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:25.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:25 smithi134 conmon[36363]: audit 2021-11-19T08:45:24.317773+0000 mon.smithi023 (mon.0) 926 : audit [DBG] from='client.? 172.21.15.23:0/3891586006' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:25.794 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.810+0000 7f87e4fb0700 1 -- 172.21.15.23:0/619033886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87e00fdd00 con 0x7f87e00f6810 2021-11-19T08:45:25.795 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.810+0000 7f87ceffd700 1 -- 172.21.15.23:0/619033886 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2577 (secure 0 0 0) 0x7f87d404e060 con 0x7f87e00f6810 2021-11-19T08:45:25.795 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:25.795 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":95,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:45:15.258264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24565,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24565":{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":94,"state":"up:resolve","state_seq":19,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:25.797 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.812+0000 7f87e4fb0700 1 -- 172.21.15.23:0/619033886 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f87c805c2a0 msgr2=0x7f87c805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:25.797 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.812+0000 7f87e4fb0700 1 --2- 172.21.15.23:0/619033886 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f87c805c2a0 0x7f87c805e760 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f87e00f6670 tx=0x7f87d0009040).stop 2021-11-19T08:45:25.797 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.812+0000 7f87e4fb0700 1 -- 172.21.15.23:0/619033886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87e00f6810 msgr2=0x7f87e010bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:25.798 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.812+0000 7f87e4fb0700 1 --2- 172.21.15.23:0/619033886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87e00f6810 0x7f87e010bc70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f87d4007a70 tx=0x7f87d4009b30).stop 2021-11-19T08:45:25.798 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.812+0000 7f87e4fb0700 1 -- 172.21.15.23:0/619033886 shutdown_connections 2021-11-19T08:45:25.798 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.812+0000 7f87e4fb0700 1 --2- 172.21.15.23:0/619033886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87e00f5570 0x7f87e0060590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:25.798 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.812+0000 7f87e4fb0700 1 --2- 172.21.15.23:0/619033886 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f87c805c2a0 0x7f87c805e760 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:25.799 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.812+0000 7f87e4fb0700 1 --2- 172.21.15.23:0/619033886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87e00f6810 0x7f87e010bc70 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:25.799 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.812+0000 7f87e4fb0700 1 -- 172.21.15.23:0/619033886 >> 172.21.15.23:0/619033886 conn(0x7f87e00f0cb0 msgr2=0x7f87e00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:25.799 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.812+0000 7f87e4fb0700 1 -- 172.21.15.23:0/619033886 shutdown_connections 2021-11-19T08:45:25.799 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:25.812+0000 7f87e4fb0700 1 -- 172.21.15.23:0/619033886 wait complete. 2021-11-19T08:45:25.800 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 95 2021-11-19T08:45:25.808 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:45:15.258264+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24565, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 94, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.134:6825/955716436', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 955716436}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 955716436}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:25.809 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:resolve 2021-11-19T08:45:26.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:26 smithi023 conmon[33490]: cluster 2021-11-19T08:45:24.905191+0000 mgr.smithi023.ehlpcl ( 2021-11-19T08:45:26.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:26 smithi023 conmon[33490]: mgr.14164) 342 : cluster [DBG] pgmap v320: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-19T08:45:26.391 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:26 smithi023 conmon[33490]: audit 2021-11-19T08:45:25.811298+0000 mon.smithi023 (mon.0) 927 : audit [DBG] 2021-11-19T08:45:26.392 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:26 smithi023 conmon[33490]: from='client.? 172.21.15.23:0/619033886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:26.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:26 smithi134 conmon[36363]: cluster 2021-11-19T08:45:24.905191+0000 mgr.smithi023.ehlpcl (mgr.14164) 342 : cluster [DBG] pgmap v320: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-11-19T08:45:26.539 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:26 smithi134 conmon[36363]: audit 2021-11-19T08:45:25.811298+0000 mon.smithi023 (mon.0) 927 : audit [DBG] from='client.? 172.21.15.23:0/619033886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:26.809 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.957+0000 7f84df388700 1 -- 172.21.15.23:0/3348699574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d80f6810 msgr2=0x7f84d80f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.957+0000 7f84df388700 1 --2- 172.21.15.23:0/3348699574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d80f6810 0x7f84d80f6c70 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f84cc004660 tx=0x7f84cc009b30).stop 2021-11-19T08:45:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.957+0000 7f84df388700 1 -- 172.21.15.23:0/3348699574 shutdown_connections 2021-11-19T08:45:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.957+0000 7f84df388700 1 --2- 172.21.15.23:0/3348699574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d80f6810 0x7f84d80f6c70 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.957+0000 7f84df388700 1 --2- 172.21.15.23:0/3348699574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84d80f5570 0x7f84d80f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.957+0000 7f84df388700 1 -- 172.21.15.23:0/3348699574 >> 172.21.15.23:0/3348699574 conn(0x7f84d80f0cb0 msgr2=0x7f84d80f30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.957+0000 7f84df388700 1 -- 172.21.15.23:0/3348699574 shutdown_connections 2021-11-19T08:45:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.957+0000 7f84df388700 1 -- 172.21.15.23:0/3348699574 wait complete. 2021-11-19T08:45:26.947 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.957+0000 7f84df388700 1 Processor -- start 2021-11-19T08:45:26.947 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.957+0000 7f84df388700 1 -- start start 2021-11-19T08:45:26.947 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.957+0000 7f84df388700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d80f5570 0x7f84d8105da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:26.947 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84df388700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84d80f6810 0x7f84d8106290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:26.948 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84df388700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d8106890 con 0x7f84d80f5570 2021-11-19T08:45:26.948 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84df388700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d81069d0 con 0x7f84d80f6810 2021-11-19T08:45:26.948 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84dd124700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d80f5570 0x7f84d8105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:26.948 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84dd124700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d80f5570 0x7f84d8105da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48976/0 (socket says 172.21.15.23:48976) 2021-11-19T08:45:26.949 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84dd124700 1 -- 172.21.15.23:0/981304172 learned_addr learned my addr 172.21.15.23:0/981304172 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:26.949 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84dc923700 1 --2- 172.21.15.23:0/981304172 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84d80f6810 0x7f84d8106290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:26.949 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84dd124700 1 -- 172.21.15.23:0/981304172 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84d80f6810 msgr2=0x7f84d8106290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:26.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84dd124700 1 --2- 172.21.15.23:0/981304172 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84d80f6810 0x7f84d8106290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:26.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84dd124700 1 -- 172.21.15.23:0/981304172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84cc005040 con 0x7f84d80f5570 2021-11-19T08:45:26.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84dd124700 1 --2- 172.21.15.23:0/981304172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d80f5570 0x7f84d8105da0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f84d0002700 tx=0x7f84d0009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:26.950 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84d5ffb700 1 -- 172.21.15.23:0/981304172 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84d00079c0 con 0x7f84d80f5570 2021-11-19T08:45:26.951 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84df388700 1 -- 172.21.15.23:0/981304172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d810b3e0 con 0x7f84d80f5570 2021-11-19T08:45:26.951 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84d5ffb700 1 -- 172.21.15.23:0/981304172 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84d000f040 con 0x7f84d80f5570 2021-11-19T08:45:26.951 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84d5ffb700 1 -- 172.21.15.23:0/981304172 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84d0014860 con 0x7f84d80f5570 2021-11-19T08:45:26.951 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.958+0000 7f84df388700 1 -- 172.21.15.23:0/981304172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d810b9a0 con 0x7f84d80f5570 2021-11-19T08:45:26.952 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.959+0000 7f84df388700 1 -- 172.21.15.23:0/981304172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d8027a00 con 0x7f84d80f5570 2021-11-19T08:45:26.952 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.960+0000 7f84d5ffb700 1 -- 172.21.15.23:0/981304172 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f84d001b060 con 0x7f84d80f5570 2021-11-19T08:45:26.952 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.960+0000 7f84d5ffb700 1 --2- 172.21.15.23:0/981304172 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f84c005c2a0 0x7f84c005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:26.952 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.960+0000 7f84d5ffb700 1 -- 172.21.15.23:0/981304172 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7f84d00762c0 con 0x7f84d80f5570 2021-11-19T08:45:26.952 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.960+0000 7f84dc923700 1 --2- 172.21.15.23:0/981304172 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f84c005c2a0 0x7f84c005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:26.953 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.960+0000 7f84dc923700 1 --2- 172.21.15.23:0/981304172 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f84c005c2a0 0x7f84c005e760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f84cc009f50 tx=0x7f84cc009b30).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:26.953 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:26.964+0000 7f84d5ffb700 1 -- 172.21.15.23:0/981304172 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84d00489d0 con 0x7f84d80f5570 2021-11-19T08:45:27.143 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:27.159+0000 7f84df388700 1 -- 172.21.15.23:0/981304172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84d80fdd00 con 0x7f84d80f5570 2021-11-19T08:45:27.144 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:27.159+0000 7f84d5ffb700 1 -- 172.21.15.23:0/981304172 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2577 (secure 0 0 0) 0x7f84d004bff0 con 0x7f84d80f5570 2021-11-19T08:45:27.144 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:27.144 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":95,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:45:15.258264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24565,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24565":{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":94,"state":"up:resolve","state_seq":19,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:27.146 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:27.161+0000 7f84df388700 1 -- 172.21.15.23:0/981304172 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f84c005c2a0 msgr2=0x7f84c005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:27.146 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:27.161+0000 7f84df388700 1 --2- 172.21.15.23:0/981304172 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f84c005c2a0 0x7f84c005e760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f84cc009f50 tx=0x7f84cc009b30).stop 2021-11-19T08:45:27.146 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:27.161+0000 7f84df388700 1 -- 172.21.15.23:0/981304172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d80f5570 msgr2=0x7f84d8105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:27.147 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:27.161+0000 7f84df388700 1 --2- 172.21.15.23:0/981304172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d80f5570 0x7f84d8105da0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f84d0002700 tx=0x7f84d0009d80).stop 2021-11-19T08:45:27.147 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:27.161+0000 7f84df388700 1 -- 172.21.15.23:0/981304172 shutdown_connections 2021-11-19T08:45:27.147 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:27.161+0000 7f84df388700 1 --2- 172.21.15.23:0/981304172 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7f84c005c2a0 0x7f84c005e760 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:27.147 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:27.161+0000 7f84df388700 1 --2- 172.21.15.23:0/981304172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d80f5570 0x7f84d8105da0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:27.148 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:27.161+0000 7f84df388700 1 --2- 172.21.15.23:0/981304172 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84d80f6810 0x7f84d8106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:27.148 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:27.161+0000 7f84df388700 1 -- 172.21.15.23:0/981304172 >> 172.21.15.23:0/981304172 conn(0x7f84d80f0cb0 msgr2=0x7f84d80fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:27.148 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:27.161+0000 7f84df388700 1 -- 172.21.15.23:0/981304172 shutdown_connections 2021-11-19T08:45:27.148 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:27.161+0000 7f84df388700 1 -- 172.21.15.23:0/981304172 wait complete. 2021-11-19T08:45:27.149 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 95 2021-11-19T08:45:27.157 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:45:15.258264+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24565, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 94, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.134:6825/955716436', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 955716436}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 955716436}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:27.158 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:resolve 2021-11-19T08:45:27.641 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:27 smithi023 conmon[33490]: audit 2021-11-19T08:45:27.160348+0000 mon.smithi023 (mon.0) 928 : audit [DBG] from='client.? 172.21.15.23:0/981304172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:27.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:27 smithi134 conmon[36363]: audit 2021-11-19T08:45:27.160348+0000 mon.smithi023 (mon.0) 928 : audit [DBG] 2021-11-19T08:45:27.789 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:27 smithi134 conmon[36363]: from='client.? 172.21.15.23:0/981304172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:28.159 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2021-11-19T08:45:28.293 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.307+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/314722118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc040f6840 msgr2=0x7fdc040f6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:28.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.307+0000 7fdc0b9d2700 1 --2- 172.21.15.23:0/314722118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc040f6840 0x7fdc040f6ca0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7fdbf4004660 tx=0x7fdbf4009b30).stop 2021-11-19T08:45:28.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.307+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/314722118 shutdown_connections 2021-11-19T08:45:28.294 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.307+0000 7fdc0b9d2700 1 --2- 172.21.15.23:0/314722118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc040f6840 0x7fdc040f6ca0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:28.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.307+0000 7fdc0b9d2700 1 --2- 172.21.15.23:0/314722118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc040f55a0 0x7fdc040f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:28.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.307+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/314722118 >> 172.21.15.23:0/314722118 conn(0x7fdc040f0ce0 msgr2=0x7fdc040f3120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:28.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.307+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/314722118 shutdown_connections 2021-11-19T08:45:28.295 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.307+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/314722118 wait complete. 2021-11-19T08:45:28.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.307+0000 7fdc0b9d2700 1 Processor -- start 2021-11-19T08:45:28.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.307+0000 7fdc0b9d2700 1 -- start start 2021-11-19T08:45:28.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdc0b9d2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc040f55a0 0x7fdc040f9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:28.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdc0b9d2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc040f6840 0x7fdc040f7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:28.296 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdc0b9d2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc040f9ad0 con 0x7fdc040f6840 2021-11-19T08:45:28.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdc0b9d2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc040f8070 con 0x7fdc040f55a0 2021-11-19T08:45:28.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdc08f6d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc040f6840 0x7fdc040f7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:28.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdc08f6d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc040f6840 0x7fdc040f7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48988/0 (socket says 172.21.15.23:48988) 2021-11-19T08:45:28.297 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdc08f6d700 1 -- 172.21.15.23:0/1278325418 learned_addr learned my addr 172.21.15.23:0/1278325418 (peer_addr_for_me v2:172.21.15.23:0/0) 2021-11-19T08:45:28.298 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdc0976e700 1 --2- 172.21.15.23:0/1278325418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc040f55a0 0x7fdc040f9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:28.298 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdc08f6d700 1 -- 172.21.15.23:0/1278325418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc040f55a0 msgr2=0x7fdc040f9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:28.298 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdc08f6d700 1 --2- 172.21.15.23:0/1278325418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc040f55a0 0x7fdc040f9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:28.298 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdc08f6d700 1 -- 172.21.15.23:0/1278325418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf4005040 con 0x7fdc040f6840 2021-11-19T08:45:28.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdc0976e700 1 --2- 172.21.15.23:0/1278325418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc040f55a0 0x7fdc040f9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-11-19T08:45:28.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdc08f6d700 1 --2- 172.21.15.23:0/1278325418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc040f6840 0x7fdc040f7b80 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fdbf4009f50 tx=0x7fdbf4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:28.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdbf9ffb700 1 -- 172.21.15.23:0/1278325418 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbf4018070 con 0x7fdc040f6840 2021-11-19T08:45:28.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/1278325418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc040f8290 con 0x7fdc040f6840 2021-11-19T08:45:28.299 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdbf9ffb700 1 -- 172.21.15.23:0/1278325418 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbf4013410 con 0x7fdc040f6840 2021-11-19T08:45:28.300 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.308+0000 7fdbf9ffb700 1 -- 172.21.15.23:0/1278325418 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbf401c3c0 con 0x7fdc040f6840 2021-11-19T08:45:28.300 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.309+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/1278325418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc040f87f0 con 0x7fdc040f6840 2021-11-19T08:45:28.301 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.310+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/1278325418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc04046330 con 0x7fdc040f6840 2021-11-19T08:45:28.302 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.310+0000 7fdbf9ffb700 1 -- 172.21.15.23:0/1278325418 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fdbf401d420 con 0x7fdc040f6840 2021-11-19T08:45:28.302 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.310+0000 7fdbf9ffb700 1 --2- 172.21.15.23:0/1278325418 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fdbf005c2a0 0x7fdbf005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-11-19T08:45:28.302 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.310+0000 7fdbf9ffb700 1 -- 172.21.15.23:0/1278325418 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6844+0+0 (secure 0 0 0) 0x7fdbf4078170 con 0x7fdc040f6840 2021-11-19T08:45:28.302 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.310+0000 7fdc0976e700 1 --2- 172.21.15.23:0/1278325418 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fdbf005c2a0 0x7fdbf005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-11-19T08:45:28.303 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.311+0000 7fdc0976e700 1 --2- 172.21.15.23:0/1278325418 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fdbf005c2a0 0x7fdbf005e760 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fdc040f66a0 tx=0x7fdbfc004d00).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-11-19T08:45:28.303 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.314+0000 7fdbf9ffb700 1 -- 172.21.15.23:0/1278325418 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdbf404a880 con 0x7fdc040f6840 2021-11-19T08:45:28.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.510+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/1278325418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc040fdd30 con 0x7fdc040f6840 2021-11-19T08:45:28.495 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.511+0000 7fdbf9ffb700 1 -- 172.21.15.23:0/1278325418 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2577 (secure 0 0 0) 0x7fdbf404dea0 con 0x7fdc040f6840 2021-11-19T08:45:28.496 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-11-19T08:45:28.496 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24673,"name":"cephfs.smithi134.opxllb","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.134:6827/2648154636","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":2648154636},{"type":"v1","addr":"172.21.15.134:6827","nonce":2648154636}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":93}],"filesystems":[{"mdsmap":{"epoch":95,"flags":50,"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2021-11-19T08:40:21.201543+0000","modified":"2021-11-19T08:45:15.258264+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":24565,"mds_1":14940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14940":{"gid":14940,"name":"cephfs.smithi023.wazanl","rank":1,"incarnation":88,"state":"up:active","state_seq":9,"addr":"172.21.15.23:6829/3287241569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3287241569},{"type":"v1","addr":"172.21.15.23:6829","nonce":3287241569}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0},"gid_24565":{"gid":24565,"name":"cephfs.smithi134.jqsnzo","rank":0,"incarnation":94,"state":"up:resolve","state_seq":19,"addr":"172.21.15.134:6825/955716436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":955716436},{"type":"v1","addr":"172.21.15.134:6825","nonce":955716436}]},"join_fscid":1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2021-11-19T08:45:28.498 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.512+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/1278325418 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fdbf005c2a0 msgr2=0x7fdbf005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:28.498 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.512+0000 7fdc0b9d2700 1 --2- 172.21.15.23:0/1278325418 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fdbf005c2a0 0x7fdbf005e760 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fdc040f66a0 tx=0x7fdbfc004d00).stop 2021-11-19T08:45:28.498 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.512+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/1278325418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc040f6840 msgr2=0x7fdc040f7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-11-19T08:45:28.498 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.512+0000 7fdc0b9d2700 1 --2- 172.21.15.23:0/1278325418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc040f6840 0x7fdc040f7b80 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fdbf4009f50 tx=0x7fdbf4009b30).stop 2021-11-19T08:45:28.499 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.512+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/1278325418 shutdown_connections 2021-11-19T08:45:28.499 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.512+0000 7fdc0b9d2700 1 --2- 172.21.15.23:0/1278325418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc040f55a0 0x7fdc040f9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:28.499 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.512+0000 7fdc0b9d2700 1 --2- 172.21.15.23:0/1278325418 >> [v2:172.21.15.23:6800/3049298848,v1:172.21.15.23:6801/3049298848] conn(0x7fdbf005c2a0 0x7fdbf005e760 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:28.499 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.513+0000 7fdc0b9d2700 1 --2- 172.21.15.23:0/1278325418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc040f6840 0x7fdc040f7b80 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-11-19T08:45:28.499 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.513+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/1278325418 >> 172.21.15.23:0/1278325418 conn(0x7fdc040f0ce0 msgr2=0x7fdc0405fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-11-19T08:45:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.513+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/1278325418 shutdown_connections 2021-11-19T08:45:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2021-11-19T08:45:28.513+0000 7fdc0b9d2700 1 -- 172.21.15.23:0/1278325418 wait complete. 2021-11-19T08:45:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 95 2021-11-19T08:45:28.508 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:28 smithi023 conmon[33490]: cluster 2021-11-19T08:45:26.905653+0000 mgr.smithi023.ehlpcl 2021-11-19T08:45:28.508 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:28 smithi023 conmon[33490]: (mgr.14164) 343 : cluster [DBG] pgmap v321: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 0 op/s 2021-11-19T08:45:28.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 50, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2021-11-19T08:40:21.201543+0000', 'modified': '2021-11-19T08:45:15.258264+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 58, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 24565, 'mds_1': 14940}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14940': {'gid': 14940, 'name': 'cephfs.smithi023.wazanl', 'rank': 1, 'incarnation': 88, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.23:6829/3287241569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6828', 'nonce': 3287241569}, {'type': 'v1', 'addr': '172.21.15.23:6829', 'nonce': 3287241569}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}, 'gid_24565': {'gid': 24565, 'name': 'cephfs.smithi134.jqsnzo', 'rank': 0, 'incarnation': 94, 'state': 'up:resolve', 'state_seq': 19, 'addr': '172.21.15.134:6825/955716436', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 955716436}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 955716436}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2021-11-19T08:45:28.510 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_24565:up:resolve 2021-11-19T08:45:28.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:28 smithi134 conmon[36363]: cluster 2021-11-19T08:45:26.905653+0000 mgr.smithi023.ehlpcl (mgr.14164) 343 : cluster [DBG] pgmap v321: 65 pgs: 65 active+clean; 5.3 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 597 B/s rd, 0 op/s 2021-11-19T08:45:29.512 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 95, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2021-11-19T08:40:21.201543+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 95, "ever_allowed_features": 32, "explicitly_allowed_features": 32, "failed": [], "flags": 50, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_14940": { "addr": "172.21.15.23:6829/3287241569", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6828", "nonce": 3287241569, "type": "v2" }, { "addr": "172.21.15.23:6829", "nonce": 3287241569, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 14940, "incarnation": 88, "join_fscid": 1, "name": "cephfs.smithi023.wazanl", "rank": 1, "state": "up:active", "state_seq": 9 }, "gid_24565": { "addr": "172.21.15.134:6825/955716436", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6824", "nonce": 955716436, "type": "v2" }, { "addr": "172.21.15.134:6825", "nonce": 955716436, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 24565, "incarnation": 94, "join_fscid": 1, "name": "cephfs.smithi134.jqsnzo", "rank": 0, "state": "up:resolve", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 2, "metadata_pool": 2, "modified": "2021-11-19T08:45:15.258264+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 1, "stopped": [], "tableserver": 0, "up": { "mds_0": 24565, "mds_1": 14940 } } } ], "standbys": [ { "addr": "172.21.15.134:6827/2648154636", "addrs": { "addrvec": [ { "addr": "172.21.15.134:6826", "nonce": 2648154636, "type": "v2" }, { "addr": "172.21.15.134:6827", "nonce": 2648154636, "type": "v1" } ] }, "epoch": 93, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 24673, "incarnation": 0, "join_fscid": 1, "name": "cephfs.smithi134.opxllb", "rank": -1, "state": "up:standby", "state_seq": 3 } ] } 2021-11-19T08:45:29.512 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e77c7dee6c987c6680b57de9907bbc4d4962f2b1/qa/tasks/kclient.py", line 112, in task kernel_mount.mount(mntopts=client_config.get('mntopts', [])) File "/home/teuthworker/src/git.ceph.com_ceph-c_e77c7dee6c987c6680b57de9907bbc4d4962f2b1/qa/tasks/cephfs/kernel_mount.py", line 44, in mount self.setupfs(name=self.cephfs_name) File "/home/teuthworker/src/git.ceph.com_ceph-c_e77c7dee6c987c6680b57de9907bbc4d4962f2b1/qa/tasks/cephfs/mount.py", line 161, in setupfs self.fs.wait_for_daemons() File "/home/teuthworker/src/git.ceph.com_ceph-c_e77c7dee6c987c6680b57de9907bbc4d4962f2b1/qa/tasks/cephfs/filesystem.py", line 1109, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2021-11-19T08:45:29.619 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2cf88e9ba7254a9482916a051a916d69 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e77c7dee6c987c6680b57de9907bbc4d4962f2b1/qa/tasks/kclient.py", line 112, in task kernel_mount.mount(mntopts=client_config.get('mntopts', [])) File "/home/teuthworker/src/git.ceph.com_ceph-c_e77c7dee6c987c6680b57de9907bbc4d4962f2b1/qa/tasks/cephfs/kernel_mount.py", line 44, in mount self.setupfs(name=self.cephfs_name) File "/home/teuthworker/src/git.ceph.com_ceph-c_e77c7dee6c987c6680b57de9907bbc4d4962f2b1/qa/tasks/cephfs/mount.py", line 161, in setupfs self.fs.wait_for_daemons() File "/home/teuthworker/src/git.ceph.com_ceph-c_e77c7dee6c987c6680b57de9907bbc4d4962f2b1/qa/tasks/cephfs/filesystem.py", line 1109, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2021-11-19T08:45:29.622 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2021-11-19T08:45:29.635 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2021-11-19T08:45:29.659 INFO:tasks.cephadm:Teardown begin 2021-11-19T08:45:29.660 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-19T08:45:29.664 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 conmon[33490]: audit 2021-11-19T08:45:28.511847+0000 mon.smithi023 (mon.0) 929 : audit [DBG] from='client.? 172.21.15.23:0/1278325418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:29.665 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 conmon[33490]: cluster 2021-11-19T08:45:29.665 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 conmon[33490]: 2021-11-19T08:45:29.247972+0000 mon.smithi023 (mon.0) 930 : cluster [WRN] Replacing daemon mds.cephfs.smithi023.wazanl as rank 1 with standby daemon mds.cephfs.smithi134.opxllb 2021-11-19T08:45:29.666 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 conmon[33490]: cluster 2021-11-19T08:45:29.248021+0000 mon.smithi023 (mon.0) 931 : cluster [INF] MDS daemon mds.cephfs.smithi023.wazanl is removed because it is dead or otherwise unavailable. 2021-11-19T08:45:29.666 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 conmon[33490]: cluster 2021-11- 2021-11-19T08:45:29.666 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 conmon[33490]: 19T08:45:29.248698+0000 mon.smithi023 (mon.0) 932 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2021-11-19T08:45:29.667 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 conmon[33490]: cluster 2021-11-19T08:45 2021-11-19T08:45:29.667 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 conmon[33490]: :29.253148+0000 mon.smithi023 (mon.0) 933 2021-11-19T08:45:29.667 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 conmon[33490]: : cluster [DBG] osdmap e59: 6 total, 6 up, 6 in 2021-11-19T08:45:29.668 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 conmon[33490]: cluster 2021-11-19T 2021-11-19T08:45:29.668 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 conmon[33490]: 08:45:29.261731+0000 mon.smithi023 (mon. 2021-11-19T08:45:29.668 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 conmon[33490]: 0) 934 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi134.opxllb=up:replay} 2021-11-19T08:45:29.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-19T08:45:29.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: audit 2021-11-19T08:45:28.511847+0000 mon.smithi023 (mon.0) 929 : audit [DBG] from='client.? 172.21.15.23:0/1278325418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2021-11-19T08:45:29.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: cluster 2021-11-19T08:45:29.247972+0000 mon.smithi023 (mon.0) 930 : cluster [WRN] Replacing daemon mds.cephfs.smithi023.wazanl as rank 1 with standby daemon mds.cephfs.smithi134.opxllb 2021-11-19T08:45:29.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: cluster 2021-11-19T08:45:29.248021+0000 mon.smithi023 (mon.0) 931 : cluster [INF] MDS daemon mds.cephfs.smithi023.wazanl is removed because it is dead or otherwise unavailable. 2021-11-19T08:45:29.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: 2021-11-19T08:45:29.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: cluster 2021-11-19T 2021-11-19T08:45:29.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: 08:45:29.248698+0000 mon.smithi023 (mon.0) 932 : 2021-11-19T08:45:29.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2021-11-19T08:45:29.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: cluster 2021-11 2021-11-19T08:45:29.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: -19T08:45:29.253148 2021-11-19T08:45:29.747 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: +0000 mon.smithi023 (mon.0) 933 : cluster 2021-11-19T08:45:29.747 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: [DBG] osdmap e59: 6 total, 6 up, 6 in 2021-11-19T08:45:29.747 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: cluster 2021-11-19 2021-11-19T08:45:29.748 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: T08:45:29.261731+0000 2021-11-19T08:45:29.748 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: mon.smithi023 (mon.0) 934 : cluster [DBG] fsmap cephfs:2/2 {0=cephfs.smithi134.jqsnzo=up:resolve,1=cephfs.smithi134.opxllb=up:replay} 2021-11-19T08:45:29.748 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:29 smithi134 conmon[36363]: 2021-11-19T08:45:29.750 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2021-11-19T08:45:29.751 DEBUG:teuthology.orchestra.run.smithi023:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2021-11-19T08:45:29.770 DEBUG:teuthology.orchestra.run.smithi134:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2021-11-19T08:45:29.808 INFO:tasks.cephadm:Stopping all daemons... 2021-11-19T08:45:29.809 INFO:tasks.cephadm.mon.smithi023:Stopping mon.smithi023... 2021-11-19T08:45:29.809 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi023 2021-11-19T08:45:29.961 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 systemd[1]: Stopping Ceph mon.smithi023 for 908b41ee-4913-11ec-8c2c-001a4aab830c... 2021-11-19T08:45:30.285 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 conmon[33490]: debug 2021-11-19T08:45:29.976+0000 7ff53482b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi023 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-19T08:45:30.286 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Nov 19 08:45:29 smithi023 conmon[33490]: debug 2021-11-19T08:45:29.976+0000 7ff53482b700 -1 mon.smithi023@0(leader) e2 *** Got Signal Terminated *** 2021-11-19T08:45:30.501 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi023.service' 2021-11-19T08:45:30.544 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-19T08:45:30.545 INFO:tasks.cephadm.mon.smithi023:Stopped mon.smithi023 2021-11-19T08:45:30.545 INFO:tasks.cephadm.mon.smithi134:Stopping mon.smithi134... 2021-11-19T08:45:30.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi134 2021-11-19T08:45:31.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:30 smithi134 systemd[1]: Stopping Ceph mon.smithi134 for 908b41ee-4913-11ec-8c2c-001a4aab830c... 2021-11-19T08:45:31.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:30 smithi134 conmon[36363]: debug 2021-11-19T08:45:30.717+0000 7f228b524700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi134 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-11-19T08:45:31.040 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:30 smithi134 conmon[36363]: debug 2021-11-19T08:45:30.717+0000 7f228b524700 -1 mon.smithi134@1(peon) e2 *** Got Signal Terminated *** 2021-11-19T08:45:31.371 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:31 smithi134 podman[59585]: ceph-908b41ee-4913-11ec-8c2c-001a4aab830c-mon.smithi134 2021-11-19T08:45:31.372 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Nov 19 08:45:31 smithi134 systemd[1]: ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi134.service: Succeeded. 2021-11-19T08:45:31.375 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@mon.smithi134.service' 2021-11-19T08:45:31.424 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-19T08:45:31.425 INFO:tasks.cephadm.mon.smithi134:Stopped mon.smithi134 2021-11-19T08:45:31.426 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2021-11-19T08:45:31.426 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.0 2021-11-19T08:45:31.812 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 19 08:45:31 smithi023 systemd[1]: Stopping Ceph osd.0 for 908b41ee-4913-11ec-8c2c-001a4aab830c... 2021-11-19T08:45:31.813 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 19 08:45:31 smithi023 conmon[50499]: debug 2021-11-19T08:45:31.567+0000 7ff64b963700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-19T08:45:31.814 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 19 08:45:31 smithi023 conmon[50499]: debug 2021-11-19T08:45:31.567+0000 7ff64b963700 -1 osd.0 59 *** Got signal Terminated *** 2021-11-19T08:45:31.814 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 19 08:45:31 smithi023 conmon[50499]: debug 2021-11-19T08:45:31.567+0000 7ff64b963700 -1 osd.0 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-19T08:45:32.113 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 19 08:45:31 smithi023 podman[96785]: ceph-908b41ee-4913-11ec-8c2c-001a4aab830c-osd.0 2021-11-19T08:45:32.391 INFO:journalctl@ceph.osd.0.smithi023.stdout:Nov 19 08:45:32 smithi023 podman[96949]: 2021-11-19T08:45:32.917 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.0.service' 2021-11-19T08:45:32.958 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-19T08:45:32.959 INFO:tasks.cephadm.osd.0:Stopped osd.0 2021-11-19T08:45:32.959 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2021-11-19T08:45:32.960 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.1 2021-11-19T08:45:33.262 INFO:journalctl@ceph.osd.1.smithi023.stdout:Nov 19 08:45:33 smithi023 systemd[1]: Stopping Ceph osd.1 for 908b41ee-4913-11ec-8c2c-001a4aab830c... 2021-11-19T08:45:33.262 INFO:journalctl@ceph.osd.1.smithi023.stdout:Nov 19 08:45:33 smithi023 conmon[55599]: debug 2021-11-19T08:45:33.139+0000 7f02de5b7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-19T08:45:33.263 INFO:journalctl@ceph.osd.1.smithi023.stdout:Nov 19 08:45:33 smithi023 conmon[55599]: debug 2021-11-19T08:45:33.139+0000 7f02de5b7700 -1 osd.1 59 *** Got signal Terminated *** 2021-11-19T08:45:33.263 INFO:journalctl@ceph.osd.1.smithi023.stdout:Nov 19 08:45:33 smithi023 conmon[55599]: debug 2021-11-19T08:45:33.139+0000 7f02de5b7700 -1 osd.1 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-19T08:45:33.641 INFO:journalctl@ceph.osd.1.smithi023.stdout:Nov 19 08:45:33 smithi023 podman[97107]: ceph-908b41ee-4913-11ec-8c2c-001a4aab830c-osd.1 2021-11-19T08:45:34.141 INFO:journalctl@ceph.osd.1.smithi023.stdout:Nov 19 08:45:33 smithi023 podman[97272]: 2021-11-19T08:45:34.488 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.1.service' 2021-11-19T08:45:34.527 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-19T08:45:34.528 INFO:tasks.cephadm.osd.1:Stopped osd.1 2021-11-19T08:45:34.528 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2021-11-19T08:45:34.529 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.2 2021-11-19T08:45:34.866 INFO:journalctl@ceph.osd.2.smithi023.stdout:Nov 19 08:45:34 smithi023 systemd[1]: Stopping Ceph osd.2 for 908b41ee-4913-11ec-8c2c-001a4aab830c... 2021-11-19T08:45:34.867 INFO:journalctl@ceph.osd.2.smithi023.stdout:Nov 19 08:45:34 smithi023 conmon[61849]: debug 2021-11-19T08:45:34.716+0000 7f98ee984700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-19T08:45:34.867 INFO:journalctl@ceph.osd.2.smithi023.stdout:Nov 19 08:45:34 smithi023 conmon[61849]: debug 2021-11-19T08:45:34.716+0000 7f98ee984700 -1 osd.2 59 *** Got signal Terminated *** 2021-11-19T08:45:34.868 INFO:journalctl@ceph.osd.2.smithi023.stdout:Nov 19 08:45:34 smithi023 conmon[61849]: debug 2021-11-19T08:45:34.716+0000 7f98ee984700 -1 osd.2 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-19T08:45:35.141 INFO:journalctl@ceph.osd.2.smithi023.stdout:Nov 19 08:45:35 smithi023 podman[97434]: ceph-908b41ee-4913-11ec-8c2c-001a4aab830c-osd.2 2021-11-19T08:45:35.641 INFO:journalctl@ceph.osd.2.smithi023.stdout:Nov 19 08:45:35 smithi023 podman[97598]: 2021-11-19T08:45:36.260 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.2.service' 2021-11-19T08:45:36.301 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-19T08:45:36.302 INFO:tasks.cephadm.osd.2:Stopped osd.2 2021-11-19T08:45:36.302 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2021-11-19T08:45:36.303 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.3 2021-11-19T08:45:36.539 INFO:journalctl@ceph.osd.3.smithi134.stdout:Nov 19 08:45:36 smithi134 systemd[1]: Stopping Ceph osd.3 for 908b41ee-4913-11ec-8c2c-001a4aab830c... 2021-11-19T08:45:36.540 INFO:journalctl@ceph.osd.3.smithi134.stdout:Nov 19 08:45:36 smithi134 conmon[42726]: debug 2021-11-19T08:45:36.457+0000 7eff57ca5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-19T08:45:36.540 INFO:journalctl@ceph.osd.3.smithi134.stdout:Nov 19 08:45:36 smithi134 conmon[42726]: debug 2021-11-19T08:45:36.457+0000 7eff57ca5700 -1 osd.3 59 *** Got signal Terminated *** 2021-11-19T08:45:36.540 INFO:journalctl@ceph.osd.3.smithi134.stdout:Nov 19 08:45:36 smithi134 conmon[42726]: debug 2021-11-19T08:45:36.457+0000 7eff57ca5700 -1 osd.3 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-19T08:45:36.986 INFO:journalctl@ceph.osd.3.smithi134.stdout:Nov 19 08:45:36 smithi134 podman[59717]: ceph-908b41ee-4913-11ec-8c2c-001a4aab830c-osd.3 2021-11-19T08:45:37.289 INFO:journalctl@ceph.osd.3.smithi134.stdout:Nov 19 08:45:37 smithi134 podman[59884]: 2021-11-19T08:45:37.997 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.3.service' 2021-11-19T08:45:38.023 INFO:journalctl@ceph.osd.3.smithi134.stdout:Nov 19 08:45:38 smithi134 systemd[1]: ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.3.service: Succeeded. 2021-11-19T08:45:38.023 INFO:journalctl@ceph.osd.3.smithi134.stdout:Nov 19 08:45:38 smithi134 systemd[1]: Stopped Ceph osd.3 for 908b41ee-4913-11ec-8c2c-001a4aab830c. 2021-11-19T08:45:38.036 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-19T08:45:38.037 INFO:tasks.cephadm.osd.3:Stopped osd.3 2021-11-19T08:45:38.037 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2021-11-19T08:45:38.038 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.4 2021-11-19T08:45:38.289 INFO:journalctl@ceph.osd.4.smithi134.stdout:Nov 19 08:45:38 smithi134 systemd[1]: Stopping Ceph osd.4 for 908b41ee-4913-11ec-8c2c-001a4aab830c... 2021-11-19T08:45:38.289 INFO:journalctl@ceph.osd.4.smithi134.stdout:Nov 19 08:45:38 smithi134 conmon[47596]: debug 2021-11-19T08:45:38.216+0000 7f58fed57700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-19T08:45:38.290 INFO:journalctl@ceph.osd.4.smithi134.stdout:Nov 19 08:45:38 smithi134 conmon[47596]: debug 2021-11-19T08:45:38.216+0000 7f58fed57700 -1 osd.4 59 *** Got signal Terminated *** 2021-11-19T08:45:38.290 INFO:journalctl@ceph.osd.4.smithi134.stdout:Nov 19 08:45:38 smithi134 conmon[47596]: debug 2021-11-19T08:45:38.216+0000 7f58fed57700 -1 osd.4 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-19T08:45:38.748 INFO:journalctl@ceph.osd.4.smithi134.stdout:Nov 19 08:45:38 smithi134 podman[60045]: ceph-908b41ee-4913-11ec-8c2c-001a4aab830c-osd.4 2021-11-19T08:45:39.039 INFO:journalctl@ceph.osd.4.smithi134.stdout:Nov 19 08:45:38 smithi134 podman[60210]: 2021-11-19T08:45:39.561 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.4.service' 2021-11-19T08:45:39.598 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-19T08:45:39.599 INFO:tasks.cephadm.osd.4:Stopped osd.4 2021-11-19T08:45:39.599 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2021-11-19T08:45:39.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.5 2021-11-19T08:45:39.900 INFO:journalctl@ceph.osd.5.smithi134.stdout:Nov 19 08:45:39 smithi134 systemd[1]: Stopping Ceph osd.5 for 908b41ee-4913-11ec-8c2c-001a4aab830c... 2021-11-19T08:45:39.901 INFO:journalctl@ceph.osd.5.smithi134.stdout:Nov 19 08:45:39 smithi134 conmon[52519]: debug 2021-11-19T08:45:39.765+0000 7f4a34254700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-11-19T08:45:39.901 INFO:journalctl@ceph.osd.5.smithi134.stdout:Nov 19 08:45:39 smithi134 conmon[52519]: debug 2021-11-19T08:45:39.765+0000 7f4a34254700 -1 osd.5 59 *** Got signal Terminated *** 2021-11-19T08:45:39.901 INFO:journalctl@ceph.osd.5.smithi134.stdout:Nov 19 08:45:39 smithi134 conmon[52519]: debug 2021-11-19T08:45:39.765+0000 7f4a34254700 -1 osd.5 59 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-11-19T08:45:40.289 INFO:journalctl@ceph.osd.5.smithi134.stdout:Nov 19 08:45:40 smithi134 podman[60369]: ceph-908b41ee-4913-11ec-8c2c-001a4aab830c-osd.5 2021-11-19T08:45:40.789 INFO:journalctl@ceph.osd.5.smithi134.stdout:Nov 19 08:45:40 smithi134 podman[60533]: 2021-11-19T08:45:41.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-908b41ee-4913-11ec-8c2c-001a4aab830c@osd.5.service' 2021-11-19T08:45:41.159 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-11-19T08:45:41.160 INFO:tasks.cephadm.osd.5:Stopped osd.5 2021-11-19T08:45:41.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c --force --keep-logs 2021-11-19T08:45:46.220 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c --force --keep-logs 2021-11-19T08:45:53.910 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-19T08:45:53.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2021-11-19T08:45:53.978 INFO:tasks.cephadm:Archiving crash dumps... 2021-11-19T08:45:53.988 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/crash to /home/teuthworker/archive/gabrioux-2021-11-19_08:00:30-rados:cephadm-wip-guits-testing-2021-11-18-1333-pacific-distro-default-smithi/6513316/remote/ubuntu@smithi023.front.sepia.ceph.com/crash 2021-11-19T08:45:53.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar cz -f - -C /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/crash -- . 2021-11-19T08:45:54.028 INFO:teuthology.orchestra.run.smithi023.stderr:tar: /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/crash: Cannot open: No such file or directory 2021-11-19T08:45:54.029 INFO:teuthology.orchestra.run.smithi023.stderr:tar: Error is not recoverable: exiting now 2021-11-19T08:45:54.031 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/crash to /home/teuthworker/archive/gabrioux-2021-11-19_08:00:30-rados:cephadm-wip-guits-testing-2021-11-18-1333-pacific-distro-default-smithi/6513316/remote/ubuntu@smithi134.front.sepia.ceph.com/crash 2021-11-19T08:45:54.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar cz -f - -C /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/crash -- . 2021-11-19T08:45:54.074 INFO:teuthology.orchestra.run.smithi134.stderr:tar: /var/lib/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/crash: Cannot open: No such file or directory 2021-11-19T08:45:54.074 INFO:teuthology.orchestra.run.smithi134.stderr:tar: Error is not recoverable: exiting now 2021-11-19T08:45:54.076 INFO:tasks.cephadm:Checking cluster log for badness... 2021-11-19T08:45:54.076 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2021-11-19T08:45:54.108 INFO:teuthology.orchestra.run.smithi023.stderr:grep: /var/log/ceph/908b41ee-4913-11ec-8c2c-001a4aab830c/ceph.log: No such file or directory 2021-11-19T08:45:54.110 INFO:tasks.cephadm:Compressing logs... 2021-11-19T08:45:54.110 DEBUG:teuthology.orchestra.run.smithi023:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-19T08:45:54.153 DEBUG:teuthology.orchestra.run.smithi134:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-19T08:45:54.178 INFO:teuthology.orchestra.run.smithi134.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2021-11-19T08:45:54.179 INFO:teuthology.orchestra.run.smithi023.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2021-11-19T08:45:55.369 INFO:tasks.cephadm:Archiving logs... 2021-11-19T08:45:55.369 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/log/ceph to /home/teuthworker/archive/gabrioux-2021-11-19_08:00:30-rados:cephadm-wip-guits-testing-2021-11-18-1333-pacific-distro-default-smithi/6513316/remote/ubuntu@smithi023.front.sepia.ceph.com/log 2021-11-19T08:45:55.371 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar cz -f - -C /var/log/ceph -- . 2021-11-19T08:45:55.703 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/gabrioux-2021-11-19_08:00:30-rados:cephadm-wip-guits-testing-2021-11-18-1333-pacific-distro-default-smithi/6513316/remote/ubuntu@smithi134.front.sepia.ceph.com/log 2021-11-19T08:45:55.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar cz -f - -C /var/log/ceph -- . 2021-11-19T08:45:55.886 INFO:tasks.cephadm:Removing cluster... 2021-11-19T08:45:55.887 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c --force 2021-11-19T08:45:56.410 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 908b41ee-4913-11ec-8c2c-001a4aab830c --force 2021-11-19T08:45:56.921 INFO:tasks.cephadm:Removing cephadm ... 2021-11-19T08:45:56.922 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/cephadm 2021-11-19T08:45:56.939 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf /home/ubuntu/cephtest/cephadm 2021-11-19T08:45:56.955 INFO:tasks.cephadm:Teardown complete 2021-11-19T08:45:56.955 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-11-19T08:45:56.969 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-11-19T08:45:56.969 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-11-19T08:45:56.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-11-19T08:45:57.068 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-11-19T08:45:57.079 INFO:teuthology.task.clock:Checking final clock skew... 2021-11-19T08:45:57.080 DEBUG:teuthology.orchestra.run.smithi023:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-19T08:45:57.083 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-11-19T08:45:57.096 INFO:teuthology.orchestra.run.smithi023.stderr:bash: ntpq: command not found 2021-11-19T08:45:57.097 INFO:teuthology.orchestra.run.smithi134.stderr:bash: ntpq: command not found 2021-11-19T08:45:57.100 INFO:teuthology.orchestra.run.smithi023.stdout:210 Number of sources = 4 2021-11-19T08:45:57.100 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-19T08:45:57.100 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2021-11-19T08:45:57.101 INFO:teuthology.orchestra.run.smithi023.stdout:^- hv01.front.sepia.ceph.com 3 6 377 64 -7793us[-7793us] +/- 133ms 2021-11-19T08:45:57.101 INFO:teuthology.orchestra.run.smithi023.stdout:^* hv02.front.sepia.ceph.com 2 6 377 65 -255us[ -217us] +/- 52ms 2021-11-19T08:45:57.101 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 -1213us[-1213us] +/- 67ms 2021-11-19T08:45:57.102 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 64 +2176us[+2176us] +/- 89ms 2021-11-19T08:45:57.104 INFO:teuthology.orchestra.run.smithi134.stdout:210 Number of sources = 4 2021-11-19T08:45:57.104 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-11-19T08:45:57.104 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2021-11-19T08:45:57.105 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 1 -6798us[-6798us] +/- 134ms 2021-11-19T08:45:57.105 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv02.front.sepia.ceph.com 2 6 377 64 +811us[ +819us] +/- 52ms 2021-11-19T08:45:57.105 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -176us[ -176us] +/- 66ms 2021-11-19T08:45:57.106 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 65 +3237us[+3245us] +/- 89ms 2021-11-19T08:45:57.106 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-11-19T08:45:57.120 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-11-19T08:45:57.121 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-11-19T08:45:57.141 DEBUG:teuthology.orchestra.run.smithi023:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-11-19T08:45:57.219 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-11-19T08:45:57.276 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-19T08:45:57.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-19T08:45:57.311 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-11-19T08:45:57.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-19T08:45:57.340 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-11-19T08:45:57.352 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-11-19T08:45:57.374 INFO:teuthology.task.internal:Duration was 1238.802928 seconds 2021-11-19T08:45:57.375 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-11-19T08:45:57.391 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-11-19T08:45:57.392 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-11-19T08:45:57.395 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-11-19T08:45:57.438 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-19T08:45:57.439 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-11-19T08:45:57.801 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-11-19T08:45:57.802 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi023.front.sepia.ceph.com 2021-11-19T08:45:57.802 DEBUG:teuthology.orchestra.run.smithi023:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-11-19T08:45:57.823 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2021-11-19T08:45:57.824 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-11-19T08:45:57.862 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-11-19T08:45:57.863 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-19T08:45:57.866 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-11-19T08:45:57.952 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2021-11-19T08:45:57.953 DEBUG:teuthology.orchestra.run.smithi023:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2021-11-19T08:45:57.956 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2021-11-19T08:45:58.116 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-11-19T08:45:58.130 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-11-19T08:45:58.131 DEBUG:teuthology.orchestra.run.smithi023:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-11-19T08:45:58.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-11-19T08:45:58.183 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-11-19T08:45:58.196 DEBUG:teuthology.orchestra.run.smithi023:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-11-19T08:45:58.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-11-19T08:45:58.223 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = core 2021-11-19T08:45:58.248 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2021-11-19T08:45:58.261 DEBUG:teuthology.orchestra.run.smithi023:> test -e /home/ubuntu/cephtest/archive/coredump 2021-11-19T08:45:58.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-19T08:45:58.292 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2021-11-19T08:45:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-11-19T08:45:58.317 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-11-19T08:45:58.330 INFO:teuthology.task.internal:Transferring archived files... 2021-11-19T08:45:58.331 DEBUG:teuthology.misc:Transferring archived files from smithi023:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2021-11-19_08:00:30-rados:cephadm-wip-guits-testing-2021-11-18-1333-pacific-distro-default-smithi/6513316/remote/smithi023 2021-11-19T08:45:58.332 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-11-19T08:45:58.389 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2021-11-19_08:00:30-rados:cephadm-wip-guits-testing-2021-11-18-1333-pacific-distro-default-smithi/6513316/remote/smithi134 2021-11-19T08:45:58.391 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-11-19T08:45:58.441 INFO:teuthology.task.internal:Removing archive directory... 2021-11-19T08:45:58.442 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf -- /home/ubuntu/cephtest/archive 2021-11-19T08:45:58.445 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2021-11-19T08:45:58.492 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-11-19T08:45:58.516 INFO:teuthology.task.internal:Not uploading archives. 2021-11-19T08:45:58.517 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-11-19T08:45:58.535 INFO:teuthology.task.internal:Tidying up after the test... 2021-11-19T08:45:58.536 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-11-19T08:45:58.538 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-11-19T08:45:58.553 INFO:teuthology.orchestra.run.smithi134.stdout: 262155 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 19 08:45 /home/ubuntu/cephtest 2021-11-19T20:22:45.707 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-11-19T20:22:45.736 DEBUG:teuthology.task.console_log:Killing console logger for smithi023 2021-11-19T20:22:45.737 DEBUG:teuthology.task.console_log:Killing console logger for smithi134 2021-11-19T20:22:45.738 DEBUG:teuthology.exit:Finished running handlers